Does Disallow in robots.txt remove a URL from Google?

No. Disallow stops compliant crawlers from fetching the URL; it does not remove the URL from Google's index. A page disallowed in robots.txt can still appear in search results based on links to it from other sites, often with a snippet that reads 'No information is available for this page.' To actually remove a page from the index, use noindex (in HTML or X-Robots-Tag) and let crawlers fetch it so they can see the noindex directive.

Should I block GPTBot, ClaudeBot, and PerplexityBot in robots.txt?

It depends on your goals. Blocking GPTBot prevents OpenAI from training future models on your content; blocking PerplexityBot or ChatGPT browsing prevents your content from being cited in live AI answers. The trade-off is real: most publishers we work with want training-blocked but live-citation-allowed, which is doable because the user agents are distinct.

Does Google honour the crawl-delay directive in 2026?

Googlebot does not honour crawl-delay and never has. Bingbot honours it. Yandex honours it. Google retired the manual Search Console crawl-rate tool in early 2024 — Google now adjusts Googlebot's crawl rate automatically based on server response signals. To slow Googlebot down temporarily, return HTTP 503 or 429 with a Retry-After header on the affected paths; for a sustained reduction, fix the underlying server-capacity issue or use the Search Console Crawl Stats report to file a special request to Google.

What is the size limit for robots.txt?

Google parses up to 500 KiB. Anything beyond that is ignored. In practice you should never approach this; most well-maintained robots.txt files are under 5 KiB.

Where does robots.txt live?

At the root of the host: https://example.com/robots.txt. Subdomains have their own robots.txt; m.example.com and www.example.com need separate files if both serve traffic.

Should I include the sitemap URL in robots.txt?

Yes. Listing your sitemap (or sitemap index) at the bottom of robots.txt is the simplest way to make sure every crawler — including ones you have not registered with — can discover it.

Can I use noindex inside robots.txt?

No. Google removed support for noindex in robots.txt in 2019. Use the meta robots tag in HTML or the X-Robots-Tag HTTP header for noindex. robots.txt is for crawl control only.

Do I need a separate robots.txt for staging?

Yes, and it should be different from production. The staging environment should have a Disallow: / blanket rule. The most common production accident in audits is the staging robots.txt shipping with the production deploy — a single line that takes the entire site out of every search index.

robots.txt for SEO in 2026: A Disciplined Guide

What robots.txt actually does (and what it does not)

robots.txt is a 30-year-old protocol with one job: tell well-behaved crawlers which URLs they may and may not fetch from your origin. It is fetched once per origin per session by compliant crawlers, parsed line-by-line, and honoured on a best-effort basis. Two things robots.txt does not do, in 2026 as in 1996: it does not remove URLs from search-engine indexes, and it does not stop non-compliant crawlers from fetching anything they want.

The persistent confusion is the difference between crawl control (what robots.txt does) and indexing control (what robots.txt does not do). A page disallowed in robots.txt that has external links pointing at it can still appear in Google with a generic snippet. The only way to remove it from the index is to allow crawl, serve a noindex directive, and wait for the crawler to fetch and process the noindex. Robots.txt alone does not get you there.

The 2026 syntax that crawlers honour

The full syntax that Googlebot honours is small. Anything beyond it is unsupported and silently ignored.

Directive	Honoured by	Notes
User-agent:	All compliant crawlers	Match is exact (case-insensitive) for the agent string. Wildcard '*' matches any agent.
Disallow:	All compliant crawlers	Path is matched as a prefix from the root, with '*' wildcard and '$' end-of-line anchors.
Allow:	Most compliant crawlers (incl. Google, Bing)	Used to carve exceptions out of a broader Disallow. Most-specific match wins.
Sitemap:	All major search engines and most AI crawlers	Absolute URL. Multiple Sitemap lines allowed; one per sitemap or sitemap index.
Crawl-delay:	Bing, Yandex (NOT Google)	Google ignores. Set crawl rate in Search Console for Google.
Host:	Yandex only	Long deprecated for Google. Safely omit.
Noindex:	Nothing in 2026	Removed from Google parsing in 2019. Use meta robots or X-Robots-Tag instead.

Wildcards: * matches any sequence of characters, $ anchors to the end of the URL. So Disallow: /*?session=$ blocks any URL ending in ?session= with no value, and Disallow: /*.pdf$ blocks every URL ending in .pdf.

Controlling AI crawlers without breaking search

The AI crawler landscape changed everything about how publishers think about robots.txt. There is now a meaningful distinction between training crawlers (which fetch your content to train future models) and live-mode crawlers (which fetch your content in real time when a user asks an AI engine a question and the engine browses the web for an answer). Most publishers we work with want different policies for each.

User agent	Operator	Purpose	Typical recommendation
GPTBot	OpenAI	Training	Block to opt out of training; allow to opt in
ChatGPT-User	OpenAI	Live browsing for ChatGPT users	Allow if you want citations in ChatGPT answers
OAI-SearchBot	OpenAI	SearchGPT indexing	Allow if you want SearchGPT visibility
ClaudeBot	Anthropic	Live browsing	Allow for Claude citations; block if you want to be invisible
anthropic-ai	Anthropic	Training (legacy/uncertain)	Conservative publishers block
PerplexityBot	Perplexity	Live citation crawling	Allow if you want Perplexity citations
Perplexity-User	Perplexity	User-initiated fetch	Allow
Google-Extended	Google	Bard/Gemini training (separate from Googlebot)	Block to opt out of Gemini training without affecting Search
Googlebot	Google	Search index	Always allow
Bingbot	Microsoft	Bing search index	Always allow
Applebot	Apple	Spotlight/Siri	Allow
Applebot-Extended	Apple	Apple Intelligence training	Block to opt out without affecting Spotlight

The distinction between training and live-mode crawlers is the single most important nuance in modern robots.txt. Block the wrong one and you make yourself invisible to the place AI users actually arrive from.

— Martin Vassilev, Founder, Toronto SEO

robots.txt patterns by site type

Lean marketing site

Allow everything to Google, Bing, and live AI crawlers. Block training crawlers if the brand wants to opt out. List sitemap. Eight to fifteen lines, no facets to worry about.

Editorial publisher

Same baseline as a marketing site, plus Disallow on /print/, /amp/ if shipped, internal search results, and any tag/date archives noindexed by other means. Strongly consider blocking training crawlers.

Ecommerce

Disallow internal search results, Disallow most faceted-navigation parameter combinations, Allow the canonical clean category URLs. Disallow /cart/, /checkout/, /account/. List the canonical product sitemap.

SaaS marketing + app

Marketing site allows everything; app subdomain (app.example.com) Disallow: / on its own robots.txt to prevent crawl of authenticated surfaces.

Multi-language

Same allow rules across language directories. Sitemap line points at the index that includes all language sitemaps. No language-specific blocks.

Headless CMS preview environments

Per-environment robots.txt enforced at the CDN. Production allows; staging and preview Disallow: /. Verify per environment after every deploy.

The robots.txt mistakes we still see in 2026

Staging robots.txt shipping to production
The single most damaging robots.txt mistake. A Disallow: / line ships, the entire site disappears from search. Add a deploy-time check that reads the production robots.txt and refuses the deploy if Disallow: / is present under User-agent: *.
Disallowing /wp-admin/ to 'protect WordPress'
Harmless but wastes a line. WordPress already serves /wp-admin/ behind authentication and the URL is not indexable anyway. Trim if you are tidying.
Disallowing /assets/ or /static/ blocking CSS and JS
Modern Google needs to fetch your CSS and JS to render the page. Blocking these breaks rendering, which can suppress ranking. Only block if you are absolutely certain crawlers do not need to render the page.
Using Disallow to try to remove a page from the index
Does not work. The page may still appear with a thin snippet, the noindex directive cannot be discovered because crawl is blocked, and the page is stuck in this state until you allow crawl.
Blocking Googlebot to 'save crawl budget' on a small site
Crawl budget concerns apply to sites with hundreds of thousands of URLs. On a 200-page marketing site, you do not have a crawl-budget problem.
Listing one Sitemap line that points to a missing or 404 sitemap
Search Console will report a sitemap error. AI crawlers that follow the Sitemap line silently fail. Test the sitemap URL after every robots.txt edit.
Using robots.txt to block faceted parameters that are linked from internal navigation
Wastes link equity flowing into URLs that are then disallowed. Better fix: stop linking the parameters internally, or canonicalize them, then optionally disallow.
Misordering Allow and Disallow assuming first-match wins
Most-specific match wins, not first-match. Test in the Google robots.txt tester or the official robotstxt parser before shipping a complex rule.

How to verify your robots.txt is doing what you think

Fetch /robots.txt from your production origin in a private browser window. Confirm the file you expect.
Run the URL Inspection tool on a sample of pages in Search Console. Confirm 'Crawl allowed: Yes' for indexable pages and 'No' for intentionally blocked ones.
Use the open-source Google robots.txt parser locally to evaluate your file against a list of test URLs.
Check the Sitemap line points at a 200-OK XML response (or sitemap index) and that every URL listed in the sitemap is allowed in robots.txt.
If you have multiple subdomains (m., shop., blog., app.), fetch each one's robots.txt independently and verify the policy makes sense for that surface.
Set a deploy-time guard: refuse any deploy where the production robots.txt contains Disallow: / under User-agent: *.
Re-run after every infrastructure change — CDN migration, host change, Cloudflare WAF rule edits can all silently change what /robots.txt returns.

The honest summary

robots.txt is small, simple, and high-stakes. The syntax is narrow; the failure modes are wide. In 2026 the file does double duty as your AI-crawler policy and your search-crawler policy, and the two policies are genuinely different. Get the basics right (allow what you want indexed, list your sitemap, block staging), decide a deliberate AI-crawler policy, and verify after every deploy. Treat the file as production code, not as configuration — because that is exactly what it is.

Frequently Asked Questions

The robots.txt questions we get asked most often during audits.