X-Robots-Tag: the noindex HTTP header, with examples

Google’s robots meta tag and X-Robots-Tag specification puts it plainly: “The X-Robots-Tag can be used as an element of the HTTP header response for a given URL. Any rule that can be used in a robots meta tag can also be specified as an X-Robots-Tag.” In a response it is one line:

X-Robots-Tag: noindex

Nothing goes in the page source. The server, or something in front of it, adds the line to the response, and a crawler reads it alongside the status code and content type. That is why it is the tool for PDFs, images and video, and for rules you want to apply across a whole site from one place.

X-Robots-Tag vs the meta robots tag

The two carry the same instructions through different channels. The meta tag lives in the HTML <head>, which is covered in our guide to the meta robots tag. The header lives in the HTTP response, so it is not HTML and does not need a <head>. Google’s wording on files is direct: “To block indexing of non-HTML resources, such as PDF files, video files, or image files, use the X-Robots-Tag response header instead.”

	Meta robots tag	X-Robots-Tag
Where it lives	HTML <head>	HTTP response header
PDFs, images, video	No	Yes
Same rules	Yes	Yes
Scope	One page per tag, or per template	Any pattern in server config, site-wide if you like
How to check	View the page source	Read the response headers
Seen if robots.txt blocks the URL	No	No

On a WordPress site the meta tag is usually set by the CMS or an SEO plugin, which our piece on WordPress noindex covers. Reach for the header when the thing you want out of the index is a file, or when a rule belongs at the server rather than in a template.

X-Robots-Tag syntax and directives

The header takes the same directives as the meta tag: noindex, nofollow, none (“Equivalent to noindex, nofollow”), nosnippet, max-snippet, max-image-preview, max-video-preview, notranslate, noimageindex, unavailable_after, indexifembedded and the default all. Google lists noarchive, nocache and nositelinkssearchbox as unused and ignores them.

An x-robots-tag noindex can be aimed at one crawler. Google: “The X-Robots-Tag may optionally specify a user agent before the rules.” Its example:

X-Robots-Tag: googlebot: nofollow
X-Robots-Tag: otherbot: noindex, nofollow

“Rules specified without a user agent are valid for all crawlers. The HTTP header, the user agent name, and the specified values are not case sensitive.” You can combine rules in a comma-separated list or send the header more than once, as Google shows with noimageindex and unavailable_after:

X-Robots-Tag: noimageindex
X-Robots-Tag: unavailable_after: 25 Jun 2010 15:00:00 PST

When rules conflict, “the more restrictive rule applies”, so nosnippet beats max-snippet:50. Other engines may not treat every rule the same way. Bing says each of its robots meta tags “can also be specified as X-Robots-Tag: in the response header”, and it still gives noarchive and nocache meanings that Google does not.

Apache and nginx examples

Google’s own examples are server configuration. On Apache they go in the site’s root .htaccess or httpd.conf; on nginx, in the site’s .conf file. Its stated benefit is that “you can specify crawling rules that are applied globally across a site”, with regular expressions doing the matching. Every PDF, Apache first, then nginx:

<Files ~ "\.pdf$">
Header set X-Robots-Tag "noindex, nofollow"
</Files>

location ~* \.pdf$ {
add_header X-Robots-Tag "noindex, nofollow";
}

For images (.png, .jpeg, .jpg, .gif), Google’s Apache rule is the same shape with a different pattern and noindex alone:

<Files ~ "\.(png|jpe?g|gif)$">
Header set X-Robots-Tag "noindex"
</Files>

The nginx version swaps in location ~* \.(png|jpe?g|gif)$ with the same add_header line. For a single file, nginx takes an exact match:

location = /secrets/unicorn.pdf {
add_header X-Robots-Tag "noindex, nofollow";
}

On Apache, a <Files "unicorn.pdf"> block does the same job, and Google’s comment on it matters: “the htaccess file must be placed in the directory of the matched file.” The pattern is the part that goes wrong. A regular expression that matches too much noindexes files you wanted found, and nothing warns you.

Setting X-Robots-Tag on Cloudflare

Cloudflare’s documentation does not mention X-Robots-Tag. What it documents is setting a response header with a Response Header Transform Rule, and X-Robots-Tag is an ordinary header name for that. Its steps: on the Rules Overview page, “Select Create rule > Response Header Transform Rule”, name the rule, choose all incoming requests or a custom filter expression, then pick an operation under Modify response header. “Set static” “Overrides the value of any existing headers with the same name or adds a new header if it does not exist.” Select Deploy to save it.

Cloudflare’s own example sets a header called X-Source on paths starting /en/. The rule below is our illustration built from those documented steps, not a Cloudflare example:

Expression:  starts_with(http.request.uri.path, "/downloads/")
Operation:   Set static
Header name: X-Robots-Tag
Value:       noindex

Two of Cloudflare’s notes apply: these rules “will also apply to default Cloudflare error pages and Custom Errors”, and “later rules can overwrite changes done by previous rules.”

X robots tag checker: how to see the header

You do not need a dedicated x robots tag checker. The header is in every response, so check what the server sends rather than what the config says. From a terminal:

curl -I https://example.com/file.pdf

curl -I sends a HEAD request, and some servers answer HEAD differently from GET. To see the headers of a real GET, use curl -s -D - -o /dev/null https://example.com/file.pdf. In a browser, open developer tools, go to the Network tab, select the request and read Response Headers.

Search Console’s URL Inspection tool shows Google’s view. Its help page says that if “Indexing allowed?” is “No”, “your site is returning a “noindex” tag or header”. To read the headers Google got, “click View crawled page”; a live test offers View tested page with “the HTTP headers” among the response data. If the noindex is deliberate and the URL shows up in the page indexing report, our guide to “Excluded by noindex tag” explains that status.

The robots.txt trap

The header has to be fetched to be obeyed. Google: “If a page is disallowed from crawling through the robots.txt file, then any information about indexing or serving rules will not be found and will therefore be ignored.” Disallow your /downloads/ folder and add a noindex header to it, and Google never sees the header.

URL Inspection hides this too. Google’s help page warns that for a URL blocked by robots.txt, “Indexing allowed? will always be “Yes” because Google can’t see and respect any noindex directives.” Read Crawl allowed? first. Our piece on crawlability vs indexability covers why the two controls are separate.

The fix is to keep the URL crawlable until the header has been seen. Google’s “Block Search indexing with noindex” page adds that “it may take months for Googlebot to revisit a page”, and that you can request a recrawl with URL Inspection. The limitation is the one every robots rule shares: it is an instruction to well-behaved crawlers, not access control, so anything that must stay private needs authentication.

Frequently asked questions

What is the X-Robots-Tag?

An HTTP response header that carries robots rules such as noindex. It accepts every rule the meta robots tag does and works on PDFs, images and other non-HTML files.

How do I set an X-Robots-Tag noindex?

Add the header in server configuration: Header set on Apache, add_header on nginx, or a Response Header Transform Rule on Cloudflare.

How do I check if a URL sends X-Robots-Tag?

Run curl -I on the URL, read Response Headers in the browser’s Network tab, or use View crawled page in Search Console’s URL Inspection.

Is X-Robots-Tag case sensitive?

No. Google says the header, the user agent name and the values are not case sensitive.

The takeaway Use the X-Robots-Tag for files and for rules set at the server. Check what the response actually contains, and never block a URL in robots.txt while you wait for its noindex to be read.