Two of the most commonly confused SEO directives are 'noindex' and 'disallow.' They look like they do the same thing — keep a page out of search results — but they work in completely different ways and are not interchangeable. Used incorrectly, they can leave URLs indexed that shouldn't be, or can permanently lock URLs into Google's index that you wanted removed. This guide covers what each does, when to use each, why combining them breaks the noindex, and the correct way to actually remove URLs from Google's index.
Noindex (in a meta robots tag or X-Robots-Tag header) tells search engines to crawl the page but not to index it. Disallow (in robots.txt) tells crawlers not to crawl the page at all. They solve different problems and combining them (disallow + noindex) breaks the noindex because the crawler never sees it. Use noindex for pages you want to remove from the index. Use disallow for pages you want to save crawl budget on. Use both only when you've already removed the URL from the index by other means.
Two of the most commonly confused SEO directives are 'noindex' and 'disallow.' They look like they do the same thing — keep a page out of search results — but they work in completely different ways and are not interchangeable. Used incorrectly, they can leave URLs indexed that shouldn't be, or can permanently lock URLs into Google's index that you wanted removed. This guide covers what each does, when to use each, why combining them breaks the noindex, and the correct way to actually remove URLs from Google's index.
Noindex is a directive placed either in the page's HTML <meta name="robots" content="noindex"> tag or in an HTTP X-Robots-Tag response header. It tells the crawler 'feel free to crawl this page, but do not include it in your index.' The crawler must access the page to see the directive — if the page is also blocked from crawling, the noindex is invisible to Google.
Noindex is the correct way to tell Google to remove a URL from search results without restricting crawl access.
Disallow is a directive in your robots.txt file that tells crawlers not to fetch a URL or URL pattern at all. It saves crawl budget by preventing crawlers from wasting requests on pages you don't want them spending time on (admin areas, infinite-scroll variations, faceted navigation that doesn't need indexing).
Critically, disallow does NOT remove a URL from Google's index. If Google already knows about the URL (via external links, sitemaps, or prior crawls) and you disallow it, Google may still index the URL — it just won't have the page content to display. You'll see indexed URLs with no description in search results, often labelled 'No information is available for this page.'
If you disallow a URL in robots.txt and add a noindex tag in the page HTML, Google can never see the noindex tag because robots.txt prevents the crawl. The page may remain in Google's index indefinitely. This is the single most common SEO mistake we audit — sites trying to 'really make sure' a page is removed by combining both directives, and accidentally locking the URL into the index.
Step 1: Add a noindex tag in the page's <meta robots> or X-Robots-Tag header. Step 2: Make sure the URL is NOT disallowed in robots.txt. Step 3: Wait for Google to recrawl the page and process the noindex. This typically takes days to weeks. Step 4 (optional, faster): Submit a removal request in Search Console's Removals tool for immediate suppression (typically 6 months) while the noindex propagates.
Once the URL has been removed from the index, you can then add a disallow if you want to save crawl budget — but only after the noindex has been processed.
Noindex: pages that exist for users but should not appear in search results. Examples: thank-you pages after form submission, internal search results, login pages, paginated archives that should not rank, A/B test variants, staging or dev URLs accidentally exposed, low-value tag/category archives.
Disallow: URL patterns that crawlers should not waste budget fetching. Examples: admin paths, infinite-scroll URL parameters, server-side script paths, faceted navigation parameters that produce duplicate content (combine with appropriate canonical strategy).
Toronto SEO works with ambitious businesses across the GTA on the exact playbooks taught in this hub. Get a free, senior-led audit of where your SEO program stands today.