Benchmarking a User Agent Parser Honestly

User agent parser benchmarks are widely published and frequently misleading. The reason is that parsing cost depends more on the corpus than on the implementation.

Matched strings are cheap and unmatched ones are not

An ordered pattern parser stops at the first match. A string matched by an early rule costs a fraction of one matched by a late rule, and an unmatched string costs the whole list.

A corpus of common browser strings therefore measures the best case. Real traffic includes crawlers, tools and unusual clients that fall much further down the list.

Throughput figures can differ by an order of magnitude between a curated corpus and a sampled one, with no change to the code.

Repeating one string measures a cache

Benchmarks that parse the same string repeatedly measure branch prediction and warm caches rather than parsing. Results look excellent and predict nothing.

Realistic input is highly repetitive but not uniformly so, with a heavy head of common strings and a long tail of rare ones.

A corpus should preserve that distribution. Deduplicating it removes exactly the property that makes caching effective in production.

Caching changes the question entirely

Because the distribution is so skewed, a result cache keyed on the string removes most of the work regardless of parser speed.

With a cache in front, parser throughput matters only for the tail. Benchmarking the parser in isolation then measures something the system rarely does.

Cache memory becomes the real constraint, since the tail of distinct strings is large and grows continuously.

Warm-up and dataset loading are often excluded

Parsers that compile patterns or build a tree at startup pay a substantial one-time cost. Benchmarks that exclude it favour designs that front-load work.

Whether that is fair depends on the deployment. A long-lived server amortises it away; a short-lived function invocation pays it on every cold start.

Stating which cost is being measured is more useful than any single number, because the right choice differs between those two cases.

Measuring what the system actually does

A useful benchmark uses sampled production strings with their real distribution, includes any cache in the measurement, and reports the cold path separately.

Accuracy should be reported alongside speed. A fast parser that misclassifies the tail is not faster in any sense the consuming system cares about.

Most teams discover the exercise is unnecessary. Parsing is rarely the bottleneck once a cache is present, and effort is better spent on dataset currency.