The crawl log is an append only table in the crawler database ], that stores when which URL was crawled, how long that crawl took and what the outcome was.
Crawl Types
| Id | Name | Description |
|---|---|---|
| 0 | file_crawl |
File crawl for general purpose indexing |
| 1 | robotstxt_fetch |
Fetching of robots.txt |
| 2 | metadata_crawl |
HEAD only request no content was indexded |