Website problems
Accidental noindex on a live page
A `noindex` directive tells search engines not to keep a page in their index. It can arrive as a `<meta name="robots" content="noindex">` tag in the HTML or as an `X-Robots-Tag: noindex` HTTP response header. Applied to a live page it removes that page from search results completely, and it produces no visible symptom: the page loads normally for every visitor. It is the single most expensive configuration mistake a site can carry, and the usual source is a staging environment whose setting was promoted to production.
The two places it lives
In the HTML: `<meta name="robots" content="noindex">` inside `<head>`. This is the visible form, and the one most people check.
In the response headers: `X-Robots-Tag: noindex`. Identical effect, applied by the web server or a reverse proxy, and invisible in view-source. This is the version that survives longest, because a developer inspecting the page sees nothing wrong.
Either can be addressed at a specific crawler — `content="googlebot: noindex"` — which makes a page indexable everywhere except where it matters.
A related directive, `nofollow`, stops links on the page being followed; it does not remove the page. `none` means both.
Why staging is almost always the cause
A staging site needs to stay out of search, and the standard way to achieve that is a site-wide `noindex`. That is correct.
It becomes a production incident when the environment variable that controls it is not set in production, when a deployment copies configuration between environments, or when the setting lives in the database and a content export carries it across.
The second most common cause is a CMS switch labelled 'discourage search engines', enabled during a build and never turned off. It is one checkbox and it applies to every page.
Both share the property that nobody notices for weeks. Traffic declines gradually as pages drop out of the index rather than disappearing on the day of the deployment.
The trap with robots.txt
Blocking a page in `robots.txt` does not remove it from the index, and it actively prevents removal.
A disallowed page is never fetched, so the crawler never sees the `noindex` on it. The URL can still be indexed from external links, and Google documents exactly this: to remove a page, crawling must be allowed and the page must serve `noindex`.
So the two mechanisms are not interchangeable. `robots.txt` controls crawling. `noindex` controls indexing. Using the first to achieve the second leaves the URL in results with no description.
How to check it yourself
`curl -I https://example.com/page` prints the response headers, which is where `X-Robots-Tag` appears. This is the check most people skip and the one that finds the hidden case.
In the browser console: `document.querySelector('meta[name=robots]')?.content` returns the meta value.
In Google Search Console, the URL inspection tool reports the indexing verdict and names the directive it found, which is the authoritative answer for that one URL.
Check a sample across templates rather than the home page alone. A `noindex` applied to one template removes a whole section while the home page looks fine.
How to fix it and confirm the fix
Find which layer emits it before changing anything. A meta tag comes from the application or the CMS; a header comes from the server, the proxy or the CDN. Editing the wrong one changes nothing and is usually diagnosed as caching.
Remove the directive for the pages that should be indexed, and keep it — deliberately — on the ones that should not: account pages, internal search results, thank-you pages.
Make sure crawling is allowed for any page you want removed from the index, so the `noindex` can actually be read.
Then request indexing for a representative URL in Search Console and watch the verdict change. Recovery is a re-crawl, not an instant switch, and a large site recovers over weeks rather than days.
Finally, make the environment difference explicit: the production build should fail, not silently inherit, when the indexing setting is absent.
How VeriFixScan detects it
`seo.noindex` reads both mechanisms for every page the crawler reaches — the robots meta tag in the HTML and the `X-Robots-Tag` response header — and reports every page that asks not to be indexed, with the URL and which of the two carried the directive.
`seo.robots_txt` covers the adjacent question of whether crawling is allowed at all, which is what determines whether a `noindex` can even be seen.
The report names pages that declare `noindex`; it does not decide whether each one should. A login page declaring it is correct, and the check reports it as a fact for you to confirm.
Frequently asked questions
- How long does it take to recover after removing noindex?
- As long as a re-crawl takes, which varies by site and by page. Google documents no guaranteed timeframe; pages crawled often return sooner, and a large site typically recovers over weeks.
- Should I use robots.txt or noindex to remove a page?
- noindex, with crawling allowed. A page blocked in robots.txt is never fetched, so its noindex is never seen, and the URL can remain indexed from external links.
- Why does my page look fine but not appear in Google?
- Check the response headers as well as the HTML. An X-Robots-Tag: noindex is invisible in view-source and behaves exactly like the meta tag.
Sources
Related
VeriFixScan crawls a site and applies its checks to every page it reaches, keeping the evidence behind each finding. Scanning one website is free.
Scan a website