Log File Analysis
Log file analysis examines the raw server access logs to see exactly which URLs Googlebot and other crawlers fetched, how often, and what status codes they received.
In context
Log file analysis is the only fully accurate way to verify what search engines actually crawl, as opposed to what they say they crawl in tools. The high-value findings: low-priority URLs eating crawl budget, important URLs Googlebot has not visited in months, status-code anomalies (sudden spikes in 5xx), bot-spoofing attempts, and which user-agent variants are most active. For Canadian enterprise sites with 100k+ URLs, log analysis is non-negotiable. Tools: Screaming Frog Log Analyzer, Botify, OnCrawl, or grep + awk on the raw logs.
A Toronto enterprise retailer's log analysis reveals Googlebot is spending 70% of crawl budget on faceted-navigation URLs that all canonicalise to the same parent — a single robots.txt block recovers the budget for product pages.
Have a question this didn't answer?
Book a free 60-minute strategy call. We'll explain how this concept applies to your specific situation.