Why Archive Crawlers Behave Unlike Search Crawlers

Archive crawlers appear in logs alongside search crawlers and are often lumped in with them. Their purpose differs, and the difference shows in what they request.

Preservation requires the whole page

An indexing crawler primarily wants text. An archive crawler needs everything required to render the page later: stylesheets, images, fonts and scripts.

Its request pattern therefore resembles a browser loading a page rather than a crawler harvesting documents, which can look unusual against expectations set by search traffic.

Bandwidth per page is correspondingly higher, and pages heavy with assets cost far more to archive than to index.

Assets loaded by script are frequently missed, which is why archived copies of script-heavy pages often render with missing images or broken layout.

Revisit schedules follow different logic

Indexing revisits are driven by how often content changes and how important the page is judged to be. Archiving revisits are driven by capturing states over time.

An archive may recapture a page that has not changed, because the point is a record at a moment rather than current content.

It may equally ignore a frequently updated page for long periods, since exhaustive currency is not the goal.

Requests may be prompted by people

Some archive fetches are triggered by someone asking for a page to be preserved. These arrive singly, without warning, and follow no crawl schedule.

Traffic that appears as a lone request from an archive operator is usually this, and is not evidence of a crawl beginning.

Rate-based rules can misread these isolated requests, particularly when they arrive for pages that receive little other traffic.

Robots handling has its own history

Archive operators have taken varying positions over time on how retrospectively robots directives should apply to material already collected.

The general direction has been toward respecting current exclusions for serving while retaining material already gathered, but practice differs by operator.

Site operators with strong views should read the specific operator's stated policy rather than assuming search crawler conventions apply.

Identifying them properly in logs

Archive crawlers announce themselves clearly, usually with a documentation address. They are easy to classify once the rules include them.

Detection rule sets aimed at search traffic often omit them, so the requests land in an unclassified bucket and get treated as anomalous.

Adding them as their own category makes both the traffic and the asset-heavy request pattern explicable rather than alarming.

It also prevents a second common error, which is counting archive fetches as visits. They are neither human traffic nor indexing, and belong in neither total.