Crawl Rate Signals and the Limits of robots.txt

Site operators worried about crawler traffic usually want to limit its rate rather than its scope. The robots file is well suited to the second problem and poorly suited to the first.

The file governs paths, not pace

Its defined directives express which paths a crawler may request. Nothing in the original convention addresses how frequently requests may be made.

That reflects the era it comes from, when the concern was crawlers reaching areas they should not rather than the load they generated.

Sites experiencing crawl load therefore reach for a tool that was designed for a different question and find it does not answer theirs.

Rate directives exist but are not universal

A delay directive is honoured by some crawlers and ignored by others, including some of the largest. It is a convention rather than part of the original specification.

Behaviour also differs between those that honour it. The value may be interpreted as a minimum gap, an average, or a hint that modifies scheduling.

Setting it is worth doing because it works for the crawlers that support it, but it cannot be relied on as the sole control.

Large operators prefer their own controls

Operators running search crawlers generally provide rate settings in their own tools, tied to a verified site ownership relationship.

That design is deliberate. It ties the request to someone who has demonstrated control of the site, which a text file cannot establish.

It also lets the operator balance the request against its own scheduling, rather than treating the file as an instruction.

Adaptive crawling responds to the server

Sophisticated crawlers adjust their rate based on observed response times and error rates. A server that slows under load is crawled less aggressively.

This means server-side rate limiting communicates effectively even without any directive, provided it returns the appropriate status rather than timing out.

Returning a clear signal that the server is temporarily overloaded is understood by well-built crawlers as a request to back off, and generally works.

What the file is genuinely good for

Excluding paths that generate expensive responses removes far more load than any rate control. Faceted listings and search endpoints are the usual offenders.

These paths often produce effectively unlimited distinct addresses, which crawlers explore thoroughly. Excluding them addresses the cause rather than the symptom.

Sites that audit what crawlers actually request usually find the load concentrated in a small number of such patterns, and the file handles that case well.