AI Search & LLMs

Do I need an llms.txt file for AI search?

Updated April 22, 2026
Quick Answer

An llms.txt file is an emerging convention that gives AI assistants a curated, machine-readable summary of your site's most important pages. Adoption by major LLM providers is partial as of April 2026 — none of the big four assistants (OpenAI, Anthropic, Google, Perplexity) have publicly committed to consuming it. We recommend publishing one anyway because the cost is trivial, several smaller agents do consume it, and adoption is trending upward. It will not move citation rankings on its own.

What llms.txt actually is

Proposed in 2024 by Jeremy Howard, llms.txt is a markdown file at your site root that lists curated URLs with one-sentence descriptions, intended as an LLM-friendly site map. A companion file, llms-full.txt, expands the markdown summaries into full content excerpts. Neither is a Google ranking factor.

Think of it as a sitemap for assistants: a curated, opinionated guide that points at your highest-value pages with context. Done well, it can influence which pages an LLM crawler prioritises if and when it reads the file.

Who actually consumes it (April 2026)

Public adoption is mixed. Several second-tier agents and developer-focused tools (Cursor, Windsurf, some open-source crawlers) consume llms.txt. The major consumer assistants have not committed publicly, though several engineers from those teams have indicated they are watching adoption.

Realistic expectation: llms.txt is a low-cost insurance policy. It will not produce measurable citation lifts in 2026, and any vendor selling 'llms.txt optimisation' as a primary ranking factor is overstating its current impact.

How to publish a useful one

Keep llms.txt to your top 30–60 most important pages. Group them by category. Write one-sentence descriptions that are genuinely useful — not marketing copy. Include the canonical URL, a short purpose, and a freshness indicator if relevant.

Generate it from the same source of truth as your sitemap so it stays current. We auto-generate ours from the site's content registry; manually maintained llms.txt files go stale within a quarter.

Want this applied to your site?

Book a free 60-minute strategy call. We'll review your site live and walk you through exactly what to fix first.

People also asked

AI Search & LLMs

How do I get cited by ChatGPT, Perplexity, and Claude?

Citations in ChatGPT, Perplexity, and Claude come from two distinct surfaces: live browsing (where the assistant runs a real-time web query and cites sources) and training data (where the model recalls facts from pre-training). For most businesses the practical lever is live browsing — earn citations by ranking well for the underlying query, structuring content into extractable answer passages of roughly 40–80 words, adding clear FAQPage and Article schema, and being mentioned on third-party sites the assistants trust (Wikipedia, established trade press, Reddit, GitHub, Stack Overflow).

Read answer
AI Search & LLMs

What is a Google AI Overview and how do I rank for one?

An AI Overview is a Gemini-generated summary that appears above organic results for many informational queries on Google. You can't 'rank' for an AIO directly — Google chooses cited sources from pages that already rank in the top organic results for the underlying query. To increase citation probability, rank in the top 5 organically, open the page with a clear answer passage, use FAQPage or HowTo schema where appropriate, and ensure the on-page content directly addresses the question Google believes the user is asking.

Read answer
AI Search & LLMs

Can I block AI from training on my content without losing AI search visibility?

Partially. The major providers separate training-data crawlers from live retrieval bots and let you opt out of one without the other. Adding GPTBot, ClaudeBot, and Google-Extended to robots.txt blocks training-data ingestion. Live citation crawlers — OAI-SearchBot, Claude-SearchBot, and Googlebot — must remain allowed for citations to keep appearing. Blocking the training crawlers has no effect on browsing-mode citations and minimal effect on existing training-data presence.

Read answer
AI Search & LLMs

Is referral traffic from ChatGPT and Perplexity actually worth anything?

Yes, disproportionately so. Across the engagements we've measured through Q1 2026, referral traffic from ChatGPT, Perplexity, and Claude converts at roughly 1.5–4× the rate of typical Google organic traffic for the same queries. Volume is still modest for most categories — usually 1–8% of total organic visits — but the per-visit value is high because the user has effectively already received an endorsement from the assistant before clicking through.

Read answer