Crawl Budget Optimization: What Google Actually Says

Crawl budget optimization is the work of making sure Googlebot spends its limited crawling time on the URLs that earn you traffic instead of duplicates, dead ends and infinite parameter variations. For most sites it is not a problem worth solving: Google's own documentation says you only need to think about it if you run roughly a million or more unique pages that change weekly, ten thousand or more pages that change daily, or you see a large share of URLs sitting in "Discovered - currently not indexed" in Search Console. If your new pages are crawled the same day you publish them, keep your sitemap current and move on.

If you are over that line, it helps to know what the budget is actually made of. Google splits it into two halves. The crawl capacity limit, sometimes called hostload, is how much simultaneous crawling your server can take without degrading. Every site starts at the same conservative default, and Google raises it automatically when responses stay fast and healthy. Latency spikes, 5xx errors and 429 responses pull it back down. The other half is crawl demand, which depends on your site's size, update frequency, page quality and relevance compared with other sites. One detail is easy to miss: demand is calculated per crawler, but the capacity limit is shared across all of Google's crawlers at once.

Google names three things that drive demand, and singles out one of them as the lever you control. Perceived inventory is the set of URLs Googlebot believes exist on your site, and Google calls it "the factor that you can positively control the most". Without guidance, Googlebot will try to crawl nearly everything it can find. Popularity and staleness make up the rest: well-linked URLs get revisited more often, and Google recrawls frequently enough to catch changes.

The most common mistake is reaching for noindex to save crawling. Google is unusually blunt about this: do not use noindex for this purpose, because Google still requests the page and only drops it after seeing the tag. The crawl has already happened by then, so you paid the full cost and saved nothing. Robots.txt is the tool that actually prevents the request. The matching caveat is that you should not use robots.txt to temporarily reallocate budget either, since freed capacity is not redirected to other pages unless you were already hitting your capacity limit.

With that cleared up, the real work is managing URL inventory. Consolidate duplicates so crawling concentrates on unique content rather than unique URLs. Block genuinely unwanted paths in robots.txt, the classic examples being sort-order variants of the same listing and infinite scroll pages that just repeat linked content. Return 404 or 410 for permanently removed pages, because a 404 is a strong signal not to crawl that URL again, while a robots-blocked URL stays in the crawl queue far longer and gets picked up again the moment the block lifts. Hunt down soft 404s too, since a page that returns 200 while saying nothing keeps getting recrawled forever.

The second category is making pages cheap to fetch. Avoid long redirect chains, which Google states have a negative effect on crawling. Improve server response times and the weight of the resources each page pulls. Support HTTP caching and answer with 304 Not Modified when nothing has changed, so unchanged pages cost almost nothing to revisit. None of this is glamorous, but it compounds across millions of requests.

Two more items are worth a line each. Keep sitemaps current and include the lastmod tag if your content changes, since Google reads them regularly to decide what deserves a look. And if you serve separate HTML to mobile and desktop, give the mobile version the same set of links. Google added this guidance in November 2024 and the reasoning is simple: only the mobile version is indexed, so a trimmed mobile navigation quietly slows discovery of everything behind it. Where parity is impossible, put those URLs in a sitemap instead.

For measurement, the URL Inspection tool reports "Hostload exceeded" when your server capacity is the binding constraint, which is the clearest signal that the fix is infrastructure rather than content. The Page Indexing report surfaces soft 404s and indexing status, and crawl stats plus raw server logs tell you which sections Googlebot visits obsessively and which it never reaches.

Finally, be realistic about raising the ceiling. Google lists only two routes: add serving capacity if you are genuinely exceeding hostload, and increase the value and uniqueness of your content. There is no setting to request more crawling. It is also worth remembering that being crawled is not the same as being indexed, since every page still has to be evaluated and assessed afterwards. Work the list in order, starting with duplicates and soft 404s, and most sites find the crawling problem was a content inventory problem all along.