How Search Engines Actually Work

Crawling is the process of discovering URLs. Google's crawler (Googlebot) starts from a queue of known URLs — typically seeded by sitemaps, internal links, external links, and previously crawled pages — and fetches the HTML of each one. The crawler then extracts new URLs from that HTML and adds them to the queue.

Crawl budget — the number of URLs Google will fetch from your site in a given window — matters for sites with tens of thousands of URLs or more. For most small-to-medium sites, crawl budget isn't the bottleneck; crawl efficiency is. Wasting crawl on faceted-navigation duplicates, soft 404s, or infinite parameter URLs is one of the most common technical SEO problems.

After fetching the raw HTML, Google renders the page using a headless Chromium instance. This means JavaScript executes, async content loads, and the rendered DOM (not just the source HTML) is what gets indexed. For modern React, Vue, and Angular sites, render is where many things go wrong.

If your critical content only appears after a JavaScript framework hydrates, Google may index it — but it costs more crawl budget and the content can lag the source. Server-side rendering (SSR), static generation, or hydration-friendly patterns dramatically improve indexing reliability for JavaScript-heavy sites.

Indexing is the process of storing the rendered content in Google's database in a structured way that supports fast retrieval. The indexer extracts entities, topics, and signals (titles, headings, structured data, internal links, anchor text) and decides whether the page is unique enough to deserve a separate entry.

Pages that are too similar to existing pages — duplicate content, near-duplicate variants, thin pages — get consolidated or dropped. The Search Console "Crawled — currently not indexed" and "Discovered — currently not indexed" reports tell you exactly where the indexer rejected your URLs and why.

When a user searches, Google's retrieval layer instantly identifies a candidate set of pages from the index that could plausibly match the query. This involves classical information-retrieval signals (term frequency, BM25), semantic embeddings, and entity matching. The retrieval layer typically pulls thousands of candidates in milliseconds.

Your goal at this stage is to make sure your pages are even in the candidate pool. That requires clear topical relevance — the right keywords, entities, and supporting content cluster — for the queries you care about.

Ranking is the final ordering of the candidate set. This is where the layered machine-learned models (RankBrain, BERT, MUM, the Helpful Content system, link-graph signals, and many others) score each candidate against the query. The ranking layer also considers user signals (clicks, dwell, pogo-sticking) over time.

Crucially, ranking is query-by-query: a page that ranks #1 for one phrasing may rank #15 for a slightly different one. That's why competitive SEO is about ranking for portfolios of related queries, not single keywords.

ChatGPT (with browsing), Claude, Perplexity, and Gemini all crawl or query the open web in some form. They retrieve candidate pages, summarize their content, and cite the sources. The same crawl/render/index/retrieve fundamentals apply, plus a few AI-specific signals: clean structured data, llms.txt files, named-entity authority, and direct, citable answers near the top of the page (TL;DRs).

Optimizing for AI citations (AEO) is not a separate discipline. It's the same SEO done with one extra question added to every page brief: would an AI assistant feel safe citing this paragraph as the answer to a user's question?