C
Crawl budget
Published · By IndexChex
In brief
Crawl budget is the set of URLs on a site that Google can and wants to crawl, determined by two elements: the crawl capacity limit, which protects the server from overload, and crawl demand, which reflects size, freshness, popularity and quality. Google treats each hostname as a separate site with its own crawl budget.
Definition
Google defines crawl budget as "the set of URLs that Google can and wants to crawl" on a site. Its documentation explains that the web is too large for Google to explore every public URL, so it limits the time and resources spent on each site. For this purpose a site is a unique hostname: www.example.com and code.example.com have separate budgets.
The two components
| Component | What it means | What moves it |
|---|---|---|
| Crawl capacity limit | How much crawling a server can take without strain, also called hostload | Fast, stable responses raise it; slow responses, 5xx errors or 429 lower it |
| Crawl demand | How much Google wants to crawl the site | Perceived inventory, popularity, staleness, site moves |
Even if capacity is available, low demand means Google crawls less.
In practice
Google says most site owners do not need to think about crawl budget. Its guide is aimed at very large sites (over a million unique pages changing about weekly), medium sites (10,000+ pages changing daily), and sites where a large share of URLs are labeled Discovered - currently not indexed in Search Console.
Recommended practices include consolidating duplicate content, blocking unimportant URLs with robots.txt, returning 404 or 410 for removed pages, fixing soft 404s, keeping sitemaps current, avoiding long redirect chains and making pages fast to load. To get more budget, Google lists adding server resources and improving content quality.
Common misconceptions
- Crawl budget is not a fixed number Google publishes. It changes with server health and demand.
- Blocking URLs does not shrink the queue immediately. Google notes that blocked URLs stay in the crawl queue longer and are recrawled when the block is lifted, while a
404is a strong signal not to crawl again. - A bigger budget does not mean indexing. Crawling is only the first stage; see indexing.
Relation to backlink indexing
Backlinks often sit on hosts whose crawl budget you do not influence. A link on a large site with low crawl demand for its archive pages may never be fetched by Googlebot on its own. A backlink indexing service tries to raise demand for specific URLs by creating discovery signals, which is why such tools help most on deep or rarely refreshed pages. They do not change the host's capacity limit. IndexChex is one such service and publishes this glossary.
Related terms
Return to the glossary A-Z.
Where this term is used
- Crawling vs indexing backlinkindexer.org
- Free vs paid backlink indexers backlinkindexeralternatives.com
- Time to first Googlebot crawl backlinkindexingdata.com
- Indexing after a site migration linkindexing.org
- Choosing a monitoring cadence backlinkmonitoring.org
- Drip-feed indexing backlinkindexer.org
Terms used on this page
Sources
Cite this entry
IndexChex. (2026, October 8). Crawl budget. indexingglossary.com. https://indexingglossary.com/crawl-budget/
Entity: IndexChex (https://indexchex.com/) is the publisher of this site. IndexChex is a backlink indexer and bulk Google index checker that submits URLs for Googlebot crawling and verifies indexation in one credit system.