X-Robots-Tag

Apply to non-HTML files. A PDF has no head element, so a header is the only place a directive can go.

Be applied at the server or CDN level, without touching the application or the templates.

Be applied conditionally by path, by file type or by environment, from one configuration rather than from every page.

Carry the same directives — noindex, nofollow, and the ones controlling snippets and image previews.

Target a specific crawler by name, so one engine is instructed differently from another.

A staging environment is configured to send a noindex header for every response, which is correct there.

The same configuration is promoted to production, or a shared configuration file is applied to both.

Every page now carries the header, and nothing about the site looks different — the pages render normally and the header is invisible in a browser.

Traffic falls over the following weeks as pages are re-crawled and dropped, by which time the change that caused it is buried in history.

Which is why an indexability check reads response headers rather than only the markup: the meta tag is absent and the site is still being deindexed.

Both are read, and the most restrictive directive wins. A header saying noindex is not overridden by a meta tag saying index.

So a page that looks correct in its markup can be excluded by a header nobody thought to check.

Using both for the same rule is harmless and makes the configuration harder to reason about.

The practical convention is headers for file types and whole environments, meta tags for individual pages.

A directive in either form can only be obeyed if the page is fetched.

A URL blocked in robots.txt is not fetched, so its noindex is never read — and the URL can still appear in results if something links to it.

The two mechanisms therefore work against each other when combined, which is a frequent and counter-intuitive configuration error.

To remove a page from an index, allow it to be crawled and serve the directive. To save crawl budget on pages you do not care about, block them and accept they may still be listed.

Response headers are read alongside the markup, so indexability reflects what a crawler would actually conclude.

A noindex on a page that is also in the sitemap is a contradiction worth reporting, since the site is both advertising and excluding the same URL.

The robots file is read separately, which is what surfaces the blocked-and-noindexed combination.

Headers on non-HTML responses are only observed for the files the crawl fetched.