How do I test my robots.txt file?

`curl -sI https://example.com/robots.txt` should return 200 with a plain text content type.

It must be at the root of the host. A file at `/subfolder/robots.txt` is ignored entirely, and so is one on a different subdomain.

The file applies per host and per scheme, so `https://www.example.com/robots.txt` and `https://example.com/robots.txt` are separate files that must both be correct.

A 404 means no restrictions and is a perfectly valid state. A 5xx or a timeout is the dangerous one: crawlers treat a persistently unreachable robots.txt as a reason to slow or stop crawling the whole host.

Check the byte size too. Google documents a 500 kibibyte parsing limit, and content past it is ignored — a file that grew past it silently loses its last rules.

Paths are matched as prefixes, not as whole path segments. `Disallow: /admin` blocks `/admin`, `/admin/users`, and also `/administration` and `/admin-guide`.

To block a directory and nothing that merely starts with the same letters, end the rule with a slash: `Disallow: /admin/`.

`$` anchors the end of a path, so `Disallow: /*.pdf$` blocks PDFs and nothing that happens to contain `.pdf` mid-path.

`*` matches any sequence of characters, which makes `Disallow: /*?sort=` a way to block a parameter wherever it appears.

When an `Allow` and a `Disallow` both match, the more specific rule wins — the longer path — rather than whichever was written first.

Rules are grouped, each group opening with one or more `User-agent` lines. A crawler uses the group that names it, and otherwise the `*` group.

It does not combine them. A crawler with its own named group ignores every rule in the wildcard group, including the ones you assumed applied to everybody.

So adding `User-agent: Googlebot` with a single `Allow: /` line silently removes Googlebot from all your `*` restrictions.

Matching is on a substring of the name and case-insensitive, so a group naming `Googlebot` also applies to `Googlebot-Image` unless a more specific group exists for it.

Search Console's robots.txt report shows the file Google last fetched, when it fetched it, and lets you test a URL against it. This is the authoritative answer for Google specifically.

It reports which line matched, which is the part that settles arguments about prefix matching.

Test the URLs you care about rather than the ones you wrote rules for: the surprise is always a path you did not think the rule reached.

Test with each user agent you named, since the group selection rule means the answer can differ per crawler.

Test the `Sitemap:` line as well by fetching the URL it names. It is read independently of the crawl rules, so a broken path there fails quietly while the rest of the file works.

It cannot remove a page from an index. A blocked URL can still be indexed from external links, appearing with no description because the content was never read.

It cannot hide anything. The file is public and lists exactly the paths you consider sensitive, which is an invitation rather than a defence.

It cannot be combined with `noindex` on the same URL: blocking the fetch means the directive is never read, and the page stays indexed.

To keep a page out of an index, allow crawling and serve `noindex`. To keep it private, require authentication.

A crawl reads the file the same way a search engine does and reports what it permits, which catches the case where the file exists and says something other than what was intended.

VeriFixScan's `seo.robots_txt` check reports whether the file is present, reachable, and what it allows for the crawl.

A rule that blocks a page the site links to from its own navigation is the finding worth acting on: the site is advertising a page it also tells crawlers to skip.

The check reports the file's state, not Google's interpretation of it. Where the two could differ, Search Console is the authority.