Concept: Crawl Data Tree

The crawl data tree is a construct in the crawler database sourrounding the crawl log.

Each crawl log entry may have:

• one mandatory crawl log entry
• multiple requests
• multiple files
  • one file text
• multiple redirects

Files and redirects reference the requst they came from if they were derived from a network request.