SEO · 7 min read
Crawl budget, and the small number of sites that need to manage it
Google defines crawl budget as the combination of how much crawling your server will tolerate and how much crawling its systems want to do. It also names the site sizes the guidance is aimed at, and most sites worrying about this are not among them.
Written by Zubair Afzal, FounderUpdated
Start here
The usual answer to "is crawl budget our problem"
Most sites asking this question do not have a crawling shortage. They have an address space that grew faster than anybody decided it should, and the fix is to stop generating addresses rather than to ask for more attention.
This is worth saying first because crawl budget is unusually easy to sell. It sounds scarce, it sounds technical, and it produces a satisfying volume of recommendations. Google’s own documentation is considerably more restrained than most of what is written about it.
The rest of this piece is the definition as published, the site profiles the guidance is written for, and a plain list of what changes crawling against what only appears to.
Definition
Two things, multiplied
Crawl budget is the combination of two separate quantities. The crawl capacity limit is described as limiting the total time a server spends holding connections open for Google, accounting for both the number of parallel connections and how long each is held. It is about what your infrastructure will tolerate.
Crawl demand is the other half, and it is about how much crawling is wanted. For Googlebot, the documentation says demand varies with a site’s size, how often it updates, page quality, and relevance compared with other sites. It is a judgement about you rather than a constraint imposed by you.
The distinction is the practical part. A faster, more reliable server raises the ceiling. Nothing you do to your infrastructure raises demand — that follows from publishing things worth returning for. Teams that treat crawl budget purely as a hosting question fix the half that was not limiting them.
The published thresholds
Who the guidance is written for
Google names the site profiles its crawl budget documentation addresses. These are the numbers in it.
- Unique pages, with content changing roughly weekly — the large-site profile
1,000,000+
Unique pages, with content changing roughly weekly — the large-site profile
Source: Google Search Central: managing crawl budget for large sites
- Unique pages, with very rapidly changing content — the medium-site profile
10,000+
Unique pages, with very rapidly changing content — the medium-site profile
Source: Google Search Central: managing crawl budget for large sites
- The unchanged-since response Google names as conserving crawl capacity
304
The unchanged-since response Google names as conserving crawl capacity
Source: Google Search Central: managing crawl budget for large sites
A third profile has no number attached: sites where a large share of addresses is classified in Search Console as discovered but not indexed. That one catches sites well under the page thresholds whose filters, parameters or internal search results have generated an address space out of proportion to what they publish.
The definition, drawn
What the two halves are, and which one you control
- Crawl capacity limit
- What your server tolerates: parallel connections and how long each is held open. Raised by being fast and reliable.
- Crawl demand
- How much crawling is wanted, varying with size, update frequency, page quality and relevance against other sites.
The test
How to tell whether crawling is genuinely the constraint
Five checks. If none of them is positive, the problem is somewhere else and the crawl budget recommendations will not touch it.
- Count the addresses your site can actually produce, including every filter, sort and parameter combination, and compare that with the number of pages you deliberately publish. A large ratio is the real finding.
- Read the server logs for the proportion of crawler requests spent on addresses you would not want indexed. If most of the attention is going to parameters, that is a crawl problem you created.
- Check how long a newly published page takes to be requested for the first time. Days is normal on a large site; weeks on a section you update daily is a signal.
- Look at whether server response times rise while crawling is heaviest. If they do, the capacity limit is being reached and infrastructure is the constraint rather than demand.
- Read the index coverage exclusions and their reasons rather than the totals. Discovered but not fetched points at crawling; crawled and then not indexed points at value, and they need opposite responses.
Sorting the advice
What changes crawling, and what only appears to
| Dimension | What it actually does | What it is often assumed to do |
|---|---|---|
| Adding noindex to low-value pages | Stops them being listed. The page still has to be crawled for the instruction to be read. | Frees crawling for other pages. The documentation says it does not shift crawl budget. |
| Blocking a section in robots.txt | Prevents the request entirely. Also makes any instruction on those pages unreadable. | Redirects crawling to better pages, which the documentation treats cautiously unless the capacity limit is being reached. |
| Consolidating duplicate content | Named as a positive step. Fewer near-identical addresses means less waste per crawl. | A tidiness exercise with no operational effect. |
| Returning 404 or 410 for removed pages | Named as a positive step. It tells the crawler to stop returning. | Something to avoid because errors look bad in reports. |
| Keeping sitemaps current with change dates | Named as a positive step, provided the dates are honest. | A way to force crawling of anything you list. |
| Improving page load speed | Named as a positive step. Faster responses let more be fetched in the same connection time. | Purely a user experience or ranking matter. |
| Supporting unchanged-since responses | Named as a positive step. A 304 avoids re-sending a page that has not changed. | An obscure caching detail with no crawl relevance. |
| Removing pages from the sitemap | Removes a hint. The addresses remain discoverable through links and will still be requested. | Removes them from crawling. It does not. |
Questions
Questions asked when an audit raises this
Does my site have a crawl budget problem?
Probably not. Google’s guidance names three profiles: large sites of a million or more unique pages whose content changes about weekly, medium or larger sites of ten thousand or more with content changing daily, and sites where a large share of addresses is reported as discovered but not indexed.
If your site is none of those, the guidance says as much, and a tool reporting a crawl budget issue is generally reporting that it found a lot of addresses rather than that anything is being missed.
Does adding noindex free up crawl budget?
No. The documentation is explicit that using noindex does not shift crawl budget, because the page still has to be crawled for the instruction to be found.
Only a robots.txt disallow prevents the request happening at all, and even that is described cautiously: blocking a section does not reallocate crawling elsewhere unless the crawl capacity limit is genuinely being reached.
What actually improves crawling?
The documentation lists consolidating duplicate content, returning proper status codes for removed pages, keeping sitemaps updated with accurate change dates, improving page load speed, and supporting responses that tell a crawler nothing has changed since it last visited.
Notice what those have in common. They reduce wasted requests and make each one cheaper, rather than asking for more attention. That is the whole shape of this discipline.
What does "discovered — currently not indexed" mean?
It means Google knows the address exists and has not fetched it, or has not chosen to store it. Google names a large share of addresses in this state as one of the profiles worth investigating for crawl budget.
It is not, on its own, evidence of a crawling shortage. It is at least as often a signal about page value — a large set of near-identical or thin pages that the site generated because it could. In that case the remedy is fewer pages, not more crawling.
How would we measure this properly?
Server logs. They record what crawlers actually requested, when, how often, what status they received and how long the response took. Everything else in this area is inference.
The questions worth asking of them: what proportion of requests went to addresses you care about, how much attention went to parameters and filters, whether response times rise under crawling load, and how long it takes for a newly published page to be requested.
Should we block our filtered URLs in robots.txt?
Sometimes, and it should be a decision rather than a default. Google publishes specific guidance on faceted navigation, and the right handling depends on whether any of those combinations are genuinely worth having indexed.
The caution is that a blocked address cannot be read, so any instruction on it — a canonical, a noindex — becomes invisible. Blocking a section already in the index is a slow and untidy way to remove it, and the sequencing has to be deliberate.
Told you have a crawl budget problem
Ask how it was measured. If the answer is a crawler count rather than server logs and coverage data, it is worth a second opinion before anybody blocks a section of your site.
Last updated