HTTP client and library user agents

Not every request comes from a browser

A significant share of production traffic is machine-to-machine: API calls, webhooks, CI pipelines, cron jobs and scraping. These clients send terse, honest user agents that follow a much simpler grammar than a browser's.

curl/8.11.1
python-requests/2.32.3
PostmanRuntime/7.43.0
Go-http-client/2.0
okhttp/4.12.0
axios/1.7.9

The format is almost always product/version with nothing else, which makes these trivially parseable — and also trivially forgeable, so treat them as a hint about intent rather than a control.

Setting a useful user agent

If you operate a client that makes requests at volume, sending a descriptive user agent is basic courtesy and often the difference between being rate-limited and being left alone. A good one names the application, its version and a contact route:

User-Agent: AcmeSync/2.4 (+https://acme.example/bot; ops@acme.example)

Some APIs require it. GitHub's REST API rejects requests without a user agent entirely, and several CDNs apply stricter rate limits to default library strings than to identified ones.

What library strings tell you about your traffic

The client mix in your logs is a useful signal on its own. A surge in python-requests with no Referer usually means scraping. Steady Go-http-client traffic is often a monitoring service or a partner integration. Dalvik means a native Android app is calling you rather than a browser, and CFNetwork with a Darwin token means a native iOS app.

Frequently asked questions

Other categories