Scanners and Probes That Fill Server Logs

Any server exposed to the internet receives continuous automated requests from systems cataloguing what is reachable. Recognising these makes the rest of the log far easier to read.

Cataloguing scanners announce themselves

Organisations that survey the internet for research or commercial datasets generally identify their scanners and publish a page explaining the activity and how to opt out.

Their requests are shallow: a root path, sometimes a known metadata path, enough to characterise what is running. They do not explore the site.

Volume from any one of them is low, but the number of such systems means the aggregate is a steady background presence.

Vulnerability probes use borrowed strings

Automated attempts to find known weaknesses typically send a common browser string or a default library string rather than identifying their purpose.

They are recognisable by their request paths, which enumerate administrative endpoints, configuration files and known application locations that do not exist on most sites.

Identifying these from paths rather than from the user agent is the reliable approach, since the header is chosen to blend in.

Library defaults reveal casual automation

Scripts written quickly often leave the default string of whatever HTTP library was used. These appear constantly and represent everything from monitoring to scraping.

A default string indicates nothing about intent. It indicates only that nobody customised it, which is true of harmless and unwelcome automation alike.

Treating a library default as inherently suspicious blocks a great deal of legitimate integration traffic, including tooling built by a site's own users.

Separating noise from signal

The value of classification is that background scanning stops competing for attention with anything unusual. A log where routine traffic is labelled makes anomalies visible.

Classification should combine the claimed identity with the request pattern, since either alone misleads. A browser string requesting only administrative paths is not a browser.

Counting by category over time is more informative than examining individual requests, because meaningful changes show as shifts in the composition.

What to do about it

Cataloguing scanners can be excluded from analytics and otherwise ignored. Their requests are cheap and blocking them accomplishes little.

Probes seeking known weaknesses are worth rate limiting by pattern, since the paths are distinctive and legitimate traffic does not request them.

The main return on this work is analytical rather than protective: it removes a large volume of traffic from reporting where it would otherwise distort every total.