Bot and crawler user agents
Googlebot, Bingbot, GPTBot, ClaudeBot, PerplexityBot, SEO crawlers, uptime monitors, security scanners and headless browsers.
157 user agents
0 matching user agents ·
| User agent string | Browser | Platform | Device | Actions |
|---|
The five kinds of bot
“Bot” covers wildly different intentions, and the right response differs for each. This database tags every crawler with a category so you can treat them differently.
- Search engines — Googlebot, Bingbot, Yandex, Baidu, Applebot. You almost certainly want these.
- AI crawlers — GPTBot, ClaudeBot, PerplexityBot, Bytespider, CCBot, Google-Extended. Whether to allow them is a business decision; all of them honour
robots.txt. - SEO tools — AhrefsBot, SemrushBot, MJ12bot, DotBot. Useful if you use the tool, pure bandwidth cost if you do not.
- Monitoring — UptimeRobot, Pingdom, Lighthouse, StatusCake. Usually yours, and should never be rate-limited.
- Scanners — sqlmap, Nikto, Censys, Shodan. Reconnaissance, generally unwelcome.
Verifying a crawler is genuine
Googlebot/2.1, and scrapers routinely do. Never grant access on the strength of the string alone.
The correct verification is a reverse DNS lookup on the requesting IP, followed by a forward lookup on the resulting hostname. A genuine Googlebot resolves to *.googlebot.com or *.google.com, and the forward lookup returns the original IP. Google, Bing and OpenAI all publish their IP ranges as JSON, which is faster to check than DNS if you cache it.
Controlling AI crawlers
Every major AI crawler respects robots.txt. To opt out of training-data collection while keeping search visibility:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Bytespider
Disallow: /Google-Extended is worth understanding: it is not a crawler that fetches anything. It is a token that controls whether content already collected by Googlebot may be used for AI training, so blocking it costs you no search ranking.
Frequently asked questions
How do I verify Googlebot is really Googlebot?
Run a reverse DNS lookup on the IP. A genuine Googlebot resolves to a hostname ending in googlebot.com or google.com. Then run a forward lookup on that hostname and confirm it returns the original IP — this second step is what defeats spoofed PTR records. Google also publishes its ranges at developers.google.com/search/apis/ipranges/googlebot.json.
Should I block AI crawlers?
It depends entirely on your model. Publishers who sell access to content usually block them; businesses who want to be cited in AI answers usually do not. Note the distinction between training crawlers such as GPTBot and ClaudeBot, and retrieval agents such as ChatGPT-User and Claude-User that fetch a page because a person asked about it — blocking the latter removes you from AI-assisted answers entirely.
What is the difference between Googlebot and Google-Extended?
Googlebot crawls pages for Search. Google-Extended fetches nothing at all — it is a robots.txt control that governs whether content Googlebot already has may be used to train Gemini and related models. Blocking Google-Extended has no effect on your search ranking.
Why do bots send a Chrome user agent?
Modern crawlers render JavaScript, so they run a real headless browser and report a real Chrome version — Googlebot's evergreen string carries the current Chrome build with compatible; Googlebot/2.1 spliced in. Parsers must check for the bot token before the browser token, or every one of these gets counted as a Chrome user.
Other categories
Desktop & Laptop
Chrome, Firefox, Safari, Edge and Opera on Windows, macOS, Linux and Chrome OS.
8,989 user agentsMobile Phones
Android and iOS phones from Samsung, Apple, Google, Xiaomi, OPPO, vivo and more.
15,434 user agentsTablets
iPad, Galaxy Tab, Fire tablets, Lenovo Tab and Windows 2-in-1 devices.
2,680 user agentsSmart TVs & Streaming
Samsung Tizen, LG webOS, Android TV, Roku, Fire TV, Apple TV and Chromecast.
86 user agentsGame Consoles
PlayStation 5 and 4, Xbox Series X|S, Nintendo Switch and Steam Deck.
20 user agentsWearables & XR
Apple Watch, Wear OS smartwatches, Meta Quest, PICO and Apple Vision Pro.
13 user agents