Normalising User Agents Before Counting Them

Grouping traffic by raw user agent string produces an unusable report. Reducing each string to a small set of stable fields is what makes aggregation meaningful.

Raw strings have too many distinct values

Every combination of browser build, platform version and device model is a separate value. A modest site sees thousands of distinct strings in a day.

The distribution has a long tail, so a top-twenty listing covers a minority of traffic and the remainder is scattered across single-occurrence values.

No question anyone actually asks is answered at that granularity. The useful questions concern browser families, device classes and platforms.

Normalisation is a deliberate loss of detail

Reducing a string to a browser name, a major version and a device class discards most of what it contained. That discarding is the point.

Which fields to keep should follow from the decisions the reporting supports, not from what the parser happens to output.

Keeping every parsed field because it is available reproduces the original problem with tidier column names.

The classification must be stable over time

Aggregates are compared across periods, so the mapping from string to category must not change silently. A parser update can reclassify traffic and shift a trend.

Recording the dataset version with each period's output makes such shifts visible and lets an analyst distinguish reclassification from real movement.

Where a comparison spans an update, reprocessing the earlier period with the newer dataset is the only way to compare like with like.

Unknowns should remain a category

Unrecognised strings must aggregate into an explicit unknown bucket rather than being assigned to the nearest plausible match.

The size of that bucket is a direct measure of how current the classification is, and a growing bucket is an early warning of a format change.

Hiding unknowns inside a general category removes both the warning and any way to judge the reporting's reliability.

Normalise once, at the boundary

Doing the reduction once when the event is recorded, and storing the classification alongside the raw value, avoids every consumer parsing independently.

Consumers parsing separately drift apart as they update at different times, and two reports built from the same logs stop agreeing.

Keeping the raw string as well preserves the option to reclassify later, which is what makes reprocessing possible when the dataset improves.