User agent

A browser string is a long accretion of historical compatibility tokens: version numbers of engines the browser is not, kept because sites once tested for them. Reading it as structured data is a mistake that has been made for thirty years.

A crawler string is usually shorter and carries a URL explaining what it is, which is the convention that makes a crawler identifiable and contactable.

A scripted client sends whatever it was configured to send, which by default is the name and version of the HTTP library — `curl/8.4.0`, for example.

Browsers have been freezing and reducing these strings for years, moving capability information to the Client Hints headers instead, because the string was never a reliable description of anything.

Every group of rules in a robots.txt file opens with `User-agent:` naming the client the rules apply to. A crawler reads the file, finds the group matching its own name, and follows those rules.

`User-agent: *` addresses every crawler that has no more specific group. A crawler that finds a group naming it specifically uses that group and ignores the wildcard entirely — a detail that causes real accidents, because rules in the wildcard group are not inherited.

Matching is on a substring of the name and is case-insensitive, so a rule addressed to `Googlebot` also applies to `Googlebot-Image` unless a more specific group exists.

The whole mechanism is voluntary. A crawler that wants to ignore robots.txt simply does, and one that wants to avoid a rule can send a different name.

Serving search engine crawlers something different from what visitors receive is cloaking, and Google's spam policies name it as a violation. The test is whether the difference is deliberate and aimed at the crawler.

Legitimate variation exists — a mobile layout, a language chosen from `Accept-Language` — and it is not cloaking because it follows the request, not the identity of the requester.

Blocking by user agent as a security measure is weak: the string is chosen by the client, and anything determined enough to be a problem will send a browser string.

The practical consequence for an audit: a site that treats an unknown user agent differently reports differently than it behaves for visitors, and the difference is worth knowing about.

`curl -A "Mozilla/5.0" -I https://example.com` sends a chosen string, and comparing that answer against the default `curl/...` one shows whether the server discriminates.

In a browser console, `navigator.userAgent` returns the string the page sees, which is what client-side detection code reads.

Server logs record the string for every request, which is the only complete picture of which clients actually reach the site.

Google's list of its own crawler strings is published, and verifying a claimed Googlebot by reverse DNS rather than by the string is the documented way to be sure.