Glossary
Crawl budget
Crawl budget is the number of URLs a search engine is willing to fetch from a site over a period. Google describes it as the product of two things: the crawl capacity your server can sustain without degrading, and the crawl demand generated by how much of your content is considered worth revisiting. It is a real constraint on very large sites and essentially irrelevant on small ones, which is where most of the confusion about it comes from.
The two halves of the limit
Crawl capacity is determined by how your server responds. Fast, consistently successful responses allow more concurrent fetching; slow responses, timeouts and server errors cause the rate to be reduced.
Crawl demand is determined by popularity and staleness. URLs that change often and are linked to are revisited more; URLs that never change and nothing links to are revisited rarely.
The two multiply rather than add: a fast server hosting content nothing asks for still gets crawled little.
You can influence capacity directly, through server performance, and demand only indirectly, through the content and its links.
Which sites it actually affects
Google's documentation states that sites with fewer than a few thousand URLs are generally crawled efficiently and do not need to think about this.
It matters for large sites — tens or hundreds of thousands of URLs — and for sites that generate URLs combinatorially.
It matters for any site where a large share of the URL space is machine-generated: faceted navigation, calendars, search result pages, session parameters.
On a small site, a page that is not being indexed is almost never a crawl budget problem, and treating it as one wastes effort on the wrong cause.
What genuinely wastes it
Infinite or near-infinite URL spaces. A calendar with a next-month link generates URLs forever, and a crawler will follow them for a long time.
Faceted navigation where every combination of filters is a crawlable URL. A handful of filters produces thousands of near-identical pages.
Long redirect chains, where reaching one page costs several fetches.
Soft 404s: pages returning 200 with no content, which are fetched repeatedly because nothing says they are worthless.
Duplicate URLs reachable through parameters, which multiply every page by the number of parameter combinations in use.
How to see it
Search Console's Crawl stats report shows requests over time, average response time, and the breakdown by response code and file type.
A rising response time alongside a falling request count is capacity being reduced, and it is a server problem rather than an SEO one.
Server logs are the complete view: which URLs were requested, how often, and what they returned.
The question worth asking of any log sample is what share of the requests went to URLs you actually wanted crawled.
Frequently asked questions
- Does my site have a crawl budget problem?
- Almost certainly not if it has fewer than a few thousand URLs. Google states such sites are generally crawled efficiently.
- Does site speed affect crawl budget?
- Yes, through crawl capacity. Fast, reliable responses allow more concurrent fetching; slow responses and server errors reduce the rate.
- Does blocking URLs in robots.txt save crawl budget?
- It stops those URLs being fetched, which helps when the blocked space is genuinely worthless. It does not help pages be indexed and it discards any links pointing at them.
Sources
Related
VeriFixScan crawls a site and applies its checks to every page it reaches, keeping the evidence behind each finding. Scanning one website is free.
Scan a website