How do I verify a crawler is really Googlebot?

`User-Agent` is a request header written by the client. Any script can send Googlebot's exact string, and scrapers routinely do.

Serving privileged content to anything claiming to be Googlebot is therefore an open door, and it is a pattern that appears in real configurations more often than it should.

The reverse is also a mistake: blocking by user agent filters honest clients and not dishonest ones.

The string is useful for logging and for robots.txt group selection, both of which assume cooperation. It is not useful for a trust decision.

Google's own crawler documentation states plainly that the user agent can be spoofed and that verification is required before treating a request as genuine.

Take the connecting IP address and perform a reverse lookup: `dig -x 66.249.66.1 +short`.

The result should be a hostname in `googlebot.com` or `google.com`. A name anywhere else fails immediately.

Then resolve that hostname forward: `dig <hostname> A +short`. The original address must be in the answer.

Both halves are required. A reverse record can be set by whoever holds the address block, so the forward lookup is what makes the pair trustworthy.

This is the method Google documents, and it works without any list to keep up to date.

Cache the result per address rather than repeating two lookups on every request: the mapping is stable, and DNS latency on the request path is a real cost at crawl volume.

The same round trip verifies the other major crawlers that publish a hostname convention, so one implementation covers several.

Google publishes JSON files listing the address ranges its crawlers use, separated by purpose: the common crawlers, the user-triggered fetchers, and other Google-operated fetchers.

Matching the connecting address against the relevant file is faster than two DNS lookups and is suitable for doing on every request.

The files change, so anything using them has to refetch periodically rather than pinning a copy.

Use the file that matches what you are verifying. A user-triggered fetch comes from a different range than a scheduled crawl, and treating them as one produces false negatives.

Log analysis: separating real search engine traffic from scrapers imitating it, so crawl statistics mean something.

Rate limiting and bot management: allowing a verified crawler through a filter that refuses unverified automated traffic.

Diagnosing a crawl problem: confirming that the requests you see in logs are the crawler you think they are before drawing conclusions from them.

It is not for serving different content. Serving crawlers something different from what visitors receive is cloaking, and Google's spam policies name it as a violation.

Bing publishes a verification tool and documents the same reverse DNS approach for Bingbot.

Several AI search crawlers publish address ranges, and others publish nothing, in which case there is no way to verify them at all.

Social platform fetchers generally publish ranges for link preview crawling.

Where an operator publishes neither a range list nor a verifiable reverse DNS convention, the honest position is that the claim cannot be checked.

Treat an unverifiable crawler as ordinary automated traffic. That is not hostility, it is the only defensible default when the claim carries no evidence.

Verification happens on your server, against your logs, using the connecting address. It is not something an external scan can perform on your behalf.

What a scan does show is the other side of the same coin: whether your site refuses automated clients, and how.

VeriFixScan reports `seo.robots_txt` for the crawl rules and `infrastructure.waf` when a firewall is present, which is what explains a crawler receiving 403 where a browser receives 200.

A site that filters automated traffic aggressively will also filter legitimate crawlers, and that shows up as pages that never get fetched rather than as an error anyone sees.

The asymmetry worth remembering: a forged Googlebot string costs you nothing if you never grant it anything, and costs you everything if you do.