The crawl log is an append only table in the crawler database, that stores when which URL was crawled, how long that crawl took and what the outcome was.
Crawl Types
| Id | Name | Description |
|---|---|---|
| 0 | file_crawl |
File crawl for general purpose indexing |
| 1 | robotstxt_fetch |
Fetching of robots.txt |
| 2 | metadata_crawl |
HEAD only request no content was indexded |