Roughly 41% of Canadian SMB sites we audited in Q1 2026 still block at least one major LLM crawler (GPTBot, ClaudeBot, PerplexityBot) by accident in robots.txt — typically inherited from a CDN default.
Across 80 Canadian SMB site audits in Q1 2026, 33 (41%) had at least one LLM crawler blocked in robots.txt. The most commonly blocked were GPTBot (28 sites) and Amazonbot (24 sites). In every case the operator confirmed the block was unintentional and inherited from a Cloudflare or default-provider configuration. Sites that opened the block saw measurable LLM citation appearances within 4–8 weeks.
Methodology
robots.txt audit across 80 Canadian SMB sites, Q1 2026; follow-up citation probe at 8 weeks for the 18 sites that opened the block.
Limits of this finding
Allowing crawlers does not guarantee citation; it only removes a blocker. Citation lift was strongest on sites with original research or pillar guides already published.
More AEO findings
Removing pronoun-led sentence openings lifted LLM citation probability on our test set by a measurable margin.
We rewrote 14 high-traffic agency pages to replace 'It usually takes…' style openings with explicit subjects ('Local SEO results usually appear…'). Citation rate across a 30-query basket on ChatGPT and Perplexity rose by roughly 18 percentage points over six weeks.
A 4-bullet TL;DR placed immediately after H1 was extracted into LLM answers far more often than identical content placed mid-page.
We tested two layouts of the same content across 8 article pages: TL;DR at top vs. TL;DR after the first body section. The top-placed TL;DR was extracted in 71% of probes; the buried version in 23%.
Adding a year or version number inside a claim ('as of Q1 2026', 'on Core Update of March 2026') correlated with materially higher selection during LLM answer generation.
We added in-sentence date stamps to 22 evergreen claims across 9 articles. Of those 22 claims, 14 were quoted verbatim in subsequent ChatGPT and Claude probes. The unstamped control set (also 22 claims) had 6 verbatim quotes.
Run experiments like this on your site.
Senior-led measurement plans, hand-counted, no laundered numbers.
Book a strategy call