Published IP Ranges as Crawler Verification

Alongside reverse lookup verification, several crawler operators publish the address ranges their systems use. Checking against a published list is faster and carries its own obligations.

The list is fetched, not resolved

Operators host a document enumerating current ranges, retrieved periodically and held in memory. Verification becomes a containment check against those ranges.

This removes the per-request lookups entirely, which matters at edge volumes where two name resolutions per request are not affordable.

The containment check itself is cheap and predictable, with no dependence on external systems at request time.

Implementations usually store the ranges in a structure built for prefix matching, so the cost stays constant as the number of published ranges grows.

Freshness becomes the operator's problem

Ranges change as infrastructure moves. A stale list rejects legitimate crawler traffic from newly added ranges.

Refreshing is therefore not optional maintenance. A list fetched once at deployment degrades steadily and silently.

Failures from staleness look like a crawler behaving oddly rather than like a configuration problem, which makes them slow to diagnose.

Fetch failures need a defined behaviour

If the list cannot be retrieved, the system must decide whether to keep using the previous copy or treat everything as unverified.

Keeping the previous copy is almost always correct, because a temporary retrieval failure is far more common than a range change that matters within hours.

Falling back to no verification during an outage removes protection at a moment nobody is watching for it.

Coverage across operators is uneven

Not every crawler operator publishes ranges, and formats differ between those that do. A general implementation ends up handling several document shapes.

Smaller and specialist crawlers frequently publish nothing, leaving reverse lookup or no verification at all as the options.

Most deployments therefore verify the operators that matter to them and treat the rest as ordinary traffic, which is a reasonable allocation of effort.

Deciding which operators matter is a question about privileges rather than volume. Only crawlers granted something worth taking need verifying at all.

Combining both methods

Range checks and reverse lookups verify the same claim with different trade-offs, and using both gives a fast path with a fallback.

A practical arrangement checks ranges first and falls back to lookups when a claimed crawler arrives from outside them, which handles range changes gracefully.

Logging which method verified each request makes the fallback rate visible, and a rising rate is a clear signal the published list needs attention.