X-Robots-Tag (HTTP header)
The X-Robots-Tag is an HTTP response header that controls indexing behaviour — equivalent to the <meta name="robots"> tag but applied at the HTTP level, useful for non-HTML resources (PDFs, images).
In context
Use cases: noindex a PDF or image without being able to add HTML meta tags, apply a noindex to an entire directory via server config, vary indexing rules by user-agent or by query parameter. Common values: 'noindex', 'nofollow', 'noarchive', 'nosnippet'. Test with curl -I to confirm the header is set; Search Console URL Inspection also reports the X-Robots-Tag value seen on a URL.
A Canadian publisher needed to noindex a directory of 5000 PDF reports without modifying each PDF; an Apache .htaccess directive set X-Robots-Tag: noindex on every PDF response — done in minutes.
Have a question this didn't answer?
Book a free 60-minute strategy call. We'll explain how this concept applies to your specific situation.