Glossary
noindex
`noindex` is an instruction telling search engines not to include a page in their index. It is expressed either as a robots meta tag in the HTML head or as an `X-Robots-Tag` HTTP header, and the header form is the only option for non-HTML files. For it to have any effect the page must be crawlable: a crawler blocked by robots.txt never fetches the page and therefore never sees the instruction.
The two places it can be set
In HTML: `<meta name="robots" content="noindex">` in the head. A `name` of `googlebot` addresses one crawler specifically rather than all of them.
As a header: `X-Robots-Tag: noindex`, which works for any response including PDFs, images and generated files that have no head to put a tag in.
Both accept additional directives — `nofollow`, `noarchive`, `nosnippet`, `max-snippet` — combined with commas.
If both are present and they disagree, the most restrictive instruction applies.
`none` is shorthand for `noindex, nofollow`, and `all` is shorthand for the permissive default. Neither adds anything a explicit directive does not, and the shorthand is easier to misread.
Why it conflicts with robots.txt
A URL disallowed in robots.txt is not fetched, so its noindex is never read. The page can then be indexed from links alone, which is the exact opposite of the intent.
The correct sequence to remove a page from an index is: allow crawling, serve noindex, wait for the page to be re-crawled and dropped, and only then block it in robots.txt if you also want to stop the requests.
Blocking first is the common mistake, and it leaves the page indexed indefinitely with no description.
Google's documentation states this directly, and Search Console reports the combination as an indexing anomaly.
Where it belongs, and where it is an accident
Legitimately: internal search results, thin filtered listings, staging environments, thank-you and checkout confirmation pages, printer-friendly duplicates.
By accident: a staging configuration promoted to production. This is among the most damaging single-line mistakes a site can make, and it removes pages silently over days.
It is not the tool for duplicates that should consolidate — a canonical tag keeps the signals; noindex discards them.
It is not a security control. The page remains fully accessible to anyone with the URL.
Paginated listing pages are a judgement rather than a rule: Google has stated it treats page two onward as ordinary pages, so removing them is a decision about whether they are useful, not a correction.
How to check it
`curl -s https://example.com/page | grep -i 'name="robots"'` finds the meta tag, and `curl -I https://example.com/page | grep -i x-robots` finds the header.
Check both: a page with a clean head can still carry the directive in a header set by the server.
Check pages that render client-side by inspecting the DOM as well as the source, since a tag injected by JavaScript is honoured when the page is rendered.
Search Console's URL inspection reports the indexing decision and the reason for it, which is the authoritative view for Google specifically.
Frequently asked questions
- Why is my noindex page still in Google?
- Usually because robots.txt blocks it, so the instruction is never read. Otherwise, because the page has not yet been re-crawled.
- Should I use noindex or a canonical tag for duplicates?
- A canonical tag, when the pages should consolidate. noindex removes the page and discards its signals instead of transferring them.
- Can I noindex a PDF?
- Yes, with the X-Robots-Tag HTTP header. A non-HTML file has no head for a meta tag.
Sources
Related
VeriFixScan crawls a site and applies its checks to every page it reaches, keeping the evidence behind each finding. Scanning one website is free.
Scan a website