Before any page can rank, Google has to discover and crawl it. Most crawlability problems aren't dramatic — they're slow leaks that waste crawl budget and leave important pages unindexed. This guide covers the audit checks and the highest-leverage fixes.
Crawlability is whether and how efficiently Google can discover and fetch your URLs. The four levers: a clean robots.txt that doesn't accidentally block important pages, an accurate XML sitemap, strong internal linking that surfaces new pages, and elimination of crawl traps (faceted navigation, parameter URLs, calendar archives).
Before any page can rank, Google has to discover and crawl it. Most crawlability problems aren't dramatic — they're slow leaks that waste crawl budget and leave important pages unindexed. This guide covers the audit checks and the highest-leverage fixes.
Google finds new URLs through internal links from existing pages, external backlinks, XML sitemaps, and (less commonly) direct submission via Search Console. Internal links are the dominant discovery channel for most sites; sitemaps are the reliable backstop.
If a page has no internal links and isn't in your sitemap, it's an orphan, and Google may never find or rank it.
robots.txt sits at /robots.txt on every domain and tells crawlers which paths they may access. A single bad line — Disallow: / — can deindex an entire site overnight. A common mistake is blocking /wp-admin/ or /admin/ but accidentally blocking /admin-related-content/.
robots.txt is for crawl control, not indexing control. Pages blocked in robots.txt can still be indexed (with no snippet) if other sites link to them. Use noindex meta tags for indexing control, not robots.txt.
An XML sitemap lists every URL you want Google to know about, with last-modified dates and (optionally) priority and change frequency. For sites under 50,000 URLs, one sitemap suffices; larger sites need a sitemap index.
Submit your sitemap in Search Console and reference it in robots.txt. Keep it accurate: stale sitemaps with 404 URLs or pages you've noindexed waste crawl budget and signal poor maintenance.
Crawl traps are URL patterns that generate effectively unlimited variations without unique content. Classic examples:
Crawl budget is the number of URLs Google will fetch from your site in a given window. It matters for sites with tens of thousands of URLs or more, and for sites with crawl traps that waste budget on junk URLs. For most small-to-medium sites, crawl budget is not the bottleneck.
Audit crawl budget in Search Console under Settings → Crawl stats. If Google is crawling thousands of low-value parameter URLs and not enough of your money pages, that's a crawl-trap problem to fix.
Run a full crawl with Screaming Frog or Sitebulb monthly on critical sites, quarterly on stable ones. Compare the crawl frontier to your sitemap to find orphans and stale URLs. Check Search Console's Coverage report for indexing problems. Cross-reference your robots.txt against the actual URLs in your crawl to catch accidental blocks.
Toronto SEO works with ambitious businesses across the GTA on the exact playbooks taught in this hub. Get a free, senior-led audit of where your SEO program stands today.