Algorithms Calculating Indexiness Crawl Delay between Requests Crawl Loop Databse Batching (Pattern) Summarizing