Robots.txt Generator

Create strict rules for web crawlers to control how your site is indexed.

Quick Presets

Common configurations to get you started quickly.

Allow all bots (Default)

Block all bots (Staging sites)

Block /wp-admin/ (WordPress)

Block AI Scrapers (GPTBot, etc)

Sitemap Configuration

Help search engines find your XML sitemap.

Sitemap URL (Absolute URL)

Crawler Rules

User Agent

Crawl Delay

Generated robots.txt

Preview of your file content

# Generated by Toronto SEO Agency (torontoseo.com)

User-agent: *
Disallow: /wp-admin/

Sitemap: https://example.com/sitemap.xml

Important Instructions

Download or copy the generated text.
Save it in a file named exactly robots.txt.
Upload it to the root directory of your website.
It must be accessible at yoursite.com/robots.txt.

Why is robots.txt important?

The robots.txt file is the first place search engine crawlers look when they visit your site. It tells them which pages they should and shouldn't crawl, helping you preserve your "crawl budget" for your most important pages and keeping private directories out of the search results.

Learn more about Technical SEO