How do you calculate crawl time under rate limits?
Most sites require a delay between requests to respect robots.txt rules or avoid overloading their servers. This calculator divides your total page count by concurrency, then multiplies by the delay between requests to estimate total crawl time.
The formula is Total time = (pages ÷ concurrency) × delay (sec). For example, crawling 10,000 pages with 5 concurrent requests and a 2-second delay gives (10,000 ÷ 5) × 2 = 4,000 seconds, about 1 hour 6 minutes. Increasing concurrency shortens the time, but exceeding the target server's policy risks getting blocked — always check the terms of service and robots.txt first.
In practice, network latency, page rendering time, and retry logic on failures all add overhead, so real crawls often take longer than this estimate. Build in some buffer when planning your schedule.
Frequently Asked Questions
Yes, but if you go below the rate limit set in the site's robots.txt or terms of service, you risk IP bans or legal issues.
Only within limits that don't overload the target server. Most sites also cap the number of concurrent connections separately.
Yes. Network latency, server response time, and retry logic all add overhead, so real-world crawls usually take longer than this calculation.