Why Log Files Overcount Unique Visitors

Deriving a visitor count from server logs is a standard exercise that reliably overstates the result. Several independent causes push in the same direction.

One person produces several values

A browser update changes the version in the string, so the same person appears as two clients across the update boundary. Updates are frequent.

Opening a link from a messaging application produces an embedded browser with a different string, then the same person may reopen it in their default browser.

Each of these is one person and several distinct values, and counting distinct values counts the boundary crossings rather than the people.

Automated traffic is a large share

Crawlers, preview fetchers, monitors and scanners all appear as clients. Their share of requests on a low-traffic site can rival or exceed human traffic.

They also generate many distinct values, since every tool and library variant is separate, which inflates distinct counts more than request counts.

Filtering the well-identified ones is straightforward and still leaves automation that presents as an ordinary browser.

The string is not an identifier

The fundamental issue is that the user agent was never meant to identify a client instance. Millions of people share the most common values exactly.

So distinct strings undercount within popular configurations and overcount across time and context. The two errors do not cancel.

Combining the string with an address improves the estimate somewhat and introduces its own errors, since shared networks collapse many people into one and mobile networks split one person across many.

Requests are the honest measure

Logs record requests accurately. Every derived measure of people or sessions is an inference layered on top with substantial uncertainty.

Reporting requests, or pages served, avoids claiming precision the data cannot support and is generally sufficient for the operational questions logs are used for.

Where an audience estimate is genuinely required, a measurement approach designed for it will be more defensible than an inference from logs.

Trends survive better than levels

Because the overcounting factors are reasonably stable, a log-derived visitor count tracks direction usefully even though its absolute value is wrong.

That stability breaks whenever a browser changes what it sends, a new crawler appears, or filtering rules are updated, and the resulting step is easily mistaken for a real change.

Recording the filtering rules and dataset version alongside the figures is what makes such steps explicable afterwards.