Product Tokens and Comments in the HTTP Specification

The user agent header has a formal grammar, and knowing it explains why parsing is hard. The grammar permits far more than it constrains.

The grammar is a sequence of products

A conforming value is one or more product tokens separated by whitespace. Each product is a name, optionally followed by a slash and a version.

Nothing in the grammar says how many products may appear or what relationship holds between them. A string naming five products is as valid as one naming a single product.

This is the root of the parsing problem. The format tells a parser where the boundaries are but nothing about which side of a boundary holds the answer it wants.

Comments are free text in parentheses

Any product may be followed by a parenthesised comment. The specification treats the contents as opaque, requiring only that parentheses balance.

Opaque means exactly that: no field order, no separators, no vocabulary. Everything inside is convention established by vendors rather than anything a parser can rely on.

Comments may nest, which surprises people writing their first parser. A naive scan for the closing parenthesis finds the wrong one when nesting occurs.

The header is advisory, not authoritative

The specification frames the header as information the client chooses to volunteer for statistical purposes and tailoring. It does not present the value as a fact about the client.

Clients are explicitly permitted to omit it. A request with no user agent header is well-formed, and treating its absence as an error rejects conforming clients.

The advisory framing also anticipates that the value may not be true. Nothing in the protocol verifies any part of it, and nothing was ever designed to.

Guidance on length exists and is ignored

The specification advises keeping the value short and warns against including unnecessary detail. Real strings are long and detailed.

The advice lost to compatibility pressure, which pushed in exactly the opposite direction. Every token added for a site check made the string longer.

Length has practical costs. The header is sent on every request, and on high-volume connections that overhead accumulates in a way that structured alternatives avoid.

What conforming to the grammar buys you

A parser written strictly to the grammar will tokenise almost any real string correctly. It will also decline to tell you which token is the browser, because the grammar does not say.

Useful parsing therefore has two layers: a grammar-level split, and a knowledge layer mapping tokens to products. Only the first can be derived from the specification.

Conflating the layers is the common mistake. Code that treats the second token as the browser is encoding a convention that the format never promised to maintain.