Toronto SEO
AI Search & LLMs

Can I block AI from training on my content without losing AI search visibility?

Updated April 22, 2026
Quick Answer

Partially. The major providers separate training-data crawlers from live retrieval bots and let you opt out of one without the other. Adding GPTBot, ClaudeBot, and Google-Extended to robots.txt blocks training-data ingestion. Live citation crawlers — OAI-SearchBot, Claude-SearchBot, and Googlebot — must remain allowed for citations to keep appearing. Blocking the training crawlers has no effect on browsing-mode citations and minimal effect on existing training-data presence.

The two-bot model

Most LLM providers now publish two distinct user agents: one for training-data crawling and one for live retrieval. OpenAI uses GPTBot for training and OAI-SearchBot for live citations. Anthropic uses ClaudeBot for training and Claude-SearchBot for live citations. Google uses Google-Extended for generative AI training and standard Googlebot for live AIO citations. Perplexity uses PerplexityBot for live retrieval and has not publicly trained models on web crawl data.

Blocking the training agents in robots.txt is a one-line operation per provider and has no impact on live-retrieval citations. We recommend most B2C and content businesses leave both surfaces open; the citation traffic outweighs the training-data leakage in almost all cases.

When opting out of training makes sense

Opt out of training when your content has direct commercial value as data (paid databases, proprietary research, paywalled publications). Opt out when you have a contractual obligation that requires it (some publisher syndication agreements). Opt out when you publish under a license that's incompatible with model training.

Do not opt out reactively because you 'want to be paid.' Major providers have not committed to broad opt-in compensation models for organic web crawl, and blocking training removes you from a long-tail of citation opportunities without producing revenue in return.

What blocking does not do

Blocking GPTBot today does not remove your content from already-trained models. It only prevents future ingestion. If a model was trained on your site in 2024, it will continue to recall that content until the next training cycle replaces it. There is no mechanism to force removal of training data short of a takedown action under specific legal frameworks.

Blocking training crawlers also does not stop other companies from scraping you with custom user agents. Robots.txt is a polite request, not enforcement. Determined scrapers route around it. If preventing scraping is a real requirement, you need server-side bot detection — not just robots.txt rules.

Want this applied to your site?

Book a free 60-minute strategy call. We'll review your site live and walk you through exactly what to fix first.

Book a Free CallGet a Free SEO Audit

People also asked

AI Search & LLMs

How do I get cited by ChatGPT, Perplexity, and Claude?

Citations in ChatGPT, Perplexity, and Claude come from two distinct surfaces: live browsing (where the assistant runs a real-time web query and cites sources) and training data (where the model recalls facts from pre-training). For most businesses the practical lever is live browsing — earn citations by ranking well for the underlying query, structuring content into extractable answer passages of roughly 40–80 words, adding clear FAQPage and Article schema, and being mentioned on third-party sites the assistants trust (Wikipedia, established trade press, Reddit, GitHub, Stack Overflow).

Read answer
AI Search & LLMs

What is a Google AI Overview and how do I rank for one?

An AI Overview is a Gemini-generated summary that appears above organic results for many informational queries on Google. You can't 'rank' for an AIO directly — Google chooses cited sources from pages that already rank in the top organic results for the underlying query. To increase citation probability, rank in the top 5 organically, open the page with a clear answer passage, use FAQPage or HowTo schema where appropriate, and ensure the on-page content directly addresses the question Google believes the user is asking.

Read answer
AI Search & LLMs

Do I need an llms.txt file for AI search?

An llms.txt file is an emerging convention that gives AI assistants a curated, machine-readable summary of your site's most important pages. Adoption by major LLM providers is partial as of April 2026 — none of the big four assistants (OpenAI, Anthropic, Google, Perplexity) have publicly committed to consuming it. We recommend publishing one anyway because the cost is trivial, several smaller agents do consume it, and adoption is trending upward. It will not move citation rankings on its own.

Read answer
AI Search & LLMs

Is referral traffic from ChatGPT and Perplexity actually worth anything?

Yes, disproportionately so. Across the engagements we've measured through Q1 2026, referral traffic from ChatGPT, Perplexity, and Claude converts at roughly 1.5–4× the rate of typical Google organic traffic for the same queries. Volume is still modest for most categories — usually 1–8% of total organic visits — but the per-visit value is high because the user has effectively already received an endorsement from the assistant before clicking through.

Read answer
Back to all answers