Technical SEO

robots.txt

Quick Definition

robots.txt is a file at yoursite.com/robots.txt that tells search engine crawlers which URLs they can and cannot fetch. It controls crawling, not indexing.

In context

Use robots.txt to prevent crawl waste on faceted URLs, search results pages, and admin paths. Never use it to keep content private — blocked URLs can still appear in search results without descriptions if they have external links.

Have a question this didn't answer?

Book a free 60-minute strategy call. We'll explain how this concept applies to your specific situation.

Book a Free Call Browse 50+ Detailed Answers

Related terms

CLS (Cumulative Layout Shift)

CLS measures unexpected visual movement during page load. The 'good' threshold is under 0.1.

Rich Results Test

Google's Rich Results Test is the official tool for validating whether a page's structured data is eligible to display as a rich result in Google Search.

LCP (Largest Contentful Paint)

LCP measures how fast the largest visible element on the page loads. The 'good' threshold is under 2.5 seconds for 75% of users.

RDFa

RDFa (Resource Description Framework in Attributes) is another inline structured data format, less common than Microdata or JSON-LD.

FID (First Input Delay)

FID measured the delay between a user's first interaction and the browser's response. It was replaced by INP as a Core Web Vital in March 2024.

Lazy Loading

Lazy loading defers loading of images and iframes until they're about to enter the viewport, reducing initial page weight.

Back to all terms