Technical SEO

What is robots.txt and how do I configure it?

Updated April 19, 2026

Quick Answer

Robots.txt is a file at yoursite.com/robots.txt that tells search engine crawlers which URLs they can and cannot fetch. It's a directive, not a blocker — it affects crawling, not indexing. Use it to prevent crawl waste on faceted URLs and admin paths, never to keep sensitive content private.

A baseline robots.txt for Toronto businesses

Most small business sites need only a minimal robots.txt that allows everything and points to the sitemap:

User-agent: * Allow: / Sitemap: https://yoursite.com/sitemap.xml

Add Disallow lines for admin paths (/wp-admin/, /admin/), checkout pages, internal search results, and faceted navigation parameters.

The most common robots.txt mistakes

Disallowing /css/ or /js/ — Google needs these to render the page and judge mobile usability. Always allow CSS and JS.

Using robots.txt to hide pages from search results — robots.txt blocks crawling, not indexing. A blocked page that has external links can still appear in search results with no description. Use noindex meta tags instead.

Forgetting to update robots.txt after a site migration — old Disallow rules can quietly block your new sitemap or important sections.

Want this applied to your site?

Book a free 60-minute strategy call. We'll review your site live and walk you through exactly what to fix first.

Book a Free Call Get a Free SEO Audit

People also asked

Technical SEO

What is technical SEO and do I need it?

Technical SEO is the work of making your site easy for Google to crawl, render, and index — Core Web Vitals, mobile usability, schema markup, XML sitemaps, internal linking, and fixing crawl errors. Every site needs at least a baseline technical audit; without it, content and link work is fighting against a broken foundation.

Read answer

Technical SEO

What are Core Web Vitals and how do I improve them?

Core Web Vitals are Google's three real-user performance metrics: Largest Contentful Paint (LCP) under 2.5s, Interaction to Next Paint (INP) under 200ms, and Cumulative Layout Shift (CLS) under 0.1. They're a confirmed ranking factor and a major UX driver — passing all three on mobile typically improves both rankings and conversion rate.

Read answer

Technical SEO

What is schema markup and which schemas matter?

Schema markup (structured data) is JSON-LD code that tells Google exactly what your page is about — a product, a recipe, a service, a local business, an FAQ. The high-impact schemas for Toronto businesses are LocalBusiness, Service, FAQPage, BreadcrumbList, Review, and Article. Done right, schema unlocks rich results that lift CTR by 20–40%.

Read answer

Technical SEO

Why did my Google rankings suddenly drop?

Sudden ranking drops are almost always caused by one of five things: a Google algorithm update, a technical change on your site (accidental noindex, broken redirects, blocked crawler), lost backlinks, a new competitor outranking you, or a manual action. The diagnosis path is the same: check Search Console first, then your CMS change log.

Read answer

Back to all answers