Toronto SEO
18 AI crawlers · per-bot rationale

AI Crawler robots.txt Builder

Decide on a per-bot basis which AI crawlers can index, train, or fetch from your site — GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot, Bytespider, and 12 more. Every toggle includes the strategic implication so you don't accidentally block your future citation surface.

Per-bot controls
17 allowed · 1 blocked
  • GPTBot
    OpenAI
    Training crawler for ChatGPT models

    Blocks your content from being used to train future GPT models. Does not affect ChatGPT browsing or live citations.

    Allow
  • OAI-SearchBot
    OpenAI
    ChatGPT Search index

    If you allow this, your pages can appear as cited sources in ChatGPT's search results. Highly recommended to allow.

    Allow
  • ChatGPT-User
    OpenAI
    Real-time fetcher when a ChatGPT user clicks a link

    On-demand fetch when a user explicitly asks about your page. Allow.

    Allow
  • PerplexityBot
    Perplexity
    Index for Perplexity answers

    Required to be cited in Perplexity. Disallow only if you do not want to appear in Perplexity sources.

    Allow
  • Perplexity-User
    Perplexity
    On-demand fetcher for user queries

    Triggered when a Perplexity user references your page directly. Allow.

    Allow
  • ClaudeBot
    Anthropic
    Training crawler for Claude models

    Blocks training use. Allow if you want long-term presence in Claude's world model; block if your content is paywalled or proprietary.

    Allow
  • anthropic-ai
    Anthropic
    Legacy training crawler

    Older Anthropic UA. Mirror your ClaudeBot decision.

    Allow
  • Claude-Web
    Anthropic
    Real-time fetcher for Claude conversations

    On-demand fetch when a Claude user asks about your page. Allow.

    Allow
  • Google-Extended
    Google
    Opt-out token for Bard/Gemini training and grounding

    Disallow to keep your content out of Gemini training, while Googlebot still indexes you for Search.

    Allow
  • CCBot
    Common Crawl
    Open dataset used to train most major LLMs

    Blocks one of the largest training corpora. Aggressive block, but the cleanest single switch if you want to opt out of LLM training broadly.

    Allow
  • Bytespider
    ByteDance / TikTok
    Training crawler

    Block by default unless you target TikTok / Doubao audiences. Bytespider is widely flagged for ignoring rate limits.

    Block
  • Diffbot
    Diffbot
    Knowledge-graph crawler used by enterprise LLM tools

    Allow if you want presence in B2B AI products built on Diffbot's graph. Block otherwise.

    Allow
  • Applebot-Extended
    Apple
    Opt-out token for Apple Intelligence training

    Disallow to opt out of Apple Intelligence training, while Applebot still indexes for Spotlight and Siri.

    Allow
  • Amazonbot
    Amazon
    Alexa and Amazon AI training

    Allow unless you specifically want to be excluded from Alexa knowledge.

    Allow
  • MistralAI-User
    Mistral
    On-demand fetcher for Le Chat

    Allow to be cited in Mistral's Le Chat answers.

    Allow
  • Meta-ExternalAgent
    Meta
    Training crawler for Llama and Meta AI

    Disallow to opt out of Llama training. Does not affect Facebook link previews (those use facebookexternalhit).

    Allow
  • FacebookBot
    Meta
    Training crawler (legacy)

    Mirror Meta-ExternalAgent decision.

    Allow
  • cohere-ai
    Cohere
    Training crawler

    Block to opt out of Cohere training. Niche but increasingly relevant for enterprise AI.

    Allow
robots.txt output
Append to your existing robots.txt at the site root.
# AI crawler controls — generated by torontoseo.com/tools/ai-crawler-robots-txt
# Append this to your existing robots.txt; it does not affect Googlebot or Bingbot.

User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: anthropic-ai
Allow: /

User-agent: Claude-Web
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: CCBot
Allow: /

User-agent: Bytespider
Disallow: /

User-agent: Diffbot
Allow: /

User-agent: Applebot-Extended
Allow: /

User-agent: Amazonbot
Allow: /

User-agent: MistralAI-User
Allow: /

User-agent: Meta-ExternalAgent
Allow: /

User-agent: FacebookBot
Allow: /

User-agent: cohere-ai
Allow: /

A note on the strategic decision

Blocking AI training crawlers (GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended) keeps your content out of model weights but does not remove you from real-time citations — those use the -User and -SearchBot agents, which are separate. The right default for almost every Canadian business: allow the on-demand fetchers and search bots, decide individually on the training crawlers based on whether your content is your moat.

Blocking everything is rarely correct — the citation traffic from Perplexity, ChatGPT Search and Google AI Overviews is now meaningful and growing. The fastest way to stop showing up in those answers is an over-aggressive block list.

Read the citation tactics playbook Need a full robots.txt? →