Glossary
robots.txt
robots.txt is a plain text file at the root of a host that tells cooperating crawlers which paths they may request. It controls crawling and not indexing: a URL blocked there can still appear in search results if something links to it, because the crawler never fetched the page to discover a noindex instruction. It is also entirely public and entirely voluntary, which makes it a traffic management tool rather than a security measure.
How the file is structured
Rules are grouped, each group beginning with one or more `User-agent` lines naming the crawlers it applies to, followed by `Disallow` and `Allow` lines.
A crawler uses the group that names it specifically if one exists, and otherwise the `*` group. It does not combine them, so rules in the wildcard group are not inherited by a crawler with its own group.
Paths are matched as prefixes. `Disallow: /admin` blocks `/admin`, `/administration` and `/admin/users` alike, which is more than most people intend.
`$` anchors the end of a path and `*` matches any sequence, both supported by the major crawlers and specified in RFC 9309.
When `Allow` and `Disallow` both match, the more specific rule wins — the longer path — rather than the first one written.
Crawling is not indexing
A blocked URL is not fetched. Google's documentation is explicit that it may still be indexed on the strength of links pointing at it, appearing with no description because the content was never read.
To keep a page out of an index, allow it to be crawled and serve a `noindex` robots meta tag or header. The two instructions are contradictory when combined: blocking the URL prevents the crawler from ever seeing the noindex.
For content that must not be reachable at all, authentication is the answer. robots.txt is published for anyone to read and names exactly the paths you consider sensitive.
Blocking a URL that already has inbound links also discards those signals, which is why blocking is rarely the right response to a duplicate.
What it is genuinely useful for
Keeping crawlers out of infinite or near-infinite URL spaces: faceted navigation, calendars, session-parameterised URLs and internal search results.
Reducing load from crawling paths that have no value in search — export endpoints, print views, sorting parameters.
Pointing at the sitemap with a `Sitemap:` line, which every major crawler reads and which does not depend on the rest of the file.
It is not useful for removing an already-indexed page, for which removal tools and noindex are the correct instruments.
How to check it
Fetch `https://example.com/robots.txt` and read it. It must be at the root of the host; a file in a subdirectory is ignored entirely.
The file applies per host and per scheme, so `https://www.example.com/robots.txt` and `https://example.com/robots.txt` are separate files that must both be correct.
A 404 for robots.txt means no restrictions and is a normal, acceptable state. A 5xx or a persistent timeout is worse: crawlers treat an unreachable robots.txt as a reason to stop crawling the host.
Search Console's robots.txt report shows what Google actually fetched and how it parsed each rule, which resolves arguments about prefix matching quickly.
Frequently asked questions
- Does robots.txt stop a page being indexed?
- No. It stops the page being fetched. A blocked URL can still appear in results from links alone, without a description.
- Can I hide private pages with robots.txt?
- No. The file is publicly readable and names the paths you want avoided. Anything private needs authentication.
- Where must robots.txt be located?
- At the root of each host and scheme. A file anywhere else is ignored, and each subdomain needs its own.
Sources
Related
VeriFixScan crawls a site and applies its checks to every page it reaches, keeping the evidence behind each finding. Scanning one website is free.
Scan a website