Crawl budget: when it matters

Recognize the large-site conditions where crawl demand and capacity deserve deliberate management.

Directional evidence Additional data required Advanced

Crawl budget is the set of URLs Google can and wants to crawl on a site. Google models it through two components:

Crawl capacity limit: How much crawling the hostname can handle without being overloaded.
Crawl demand: How much Google wants to crawl based on inventory, popularity, staleness, page quality, relevance, and product-specific needs.

Most sites do not have a crawl-budget problem. A page can be crawled and still remain unindexed; more crawl requests do not guarantee indexing or ranking.

Who should investigate crawl budget

Google's current advanced guidance is aimed mainly at:

Large sites with roughly 1 million or more unique pages whose content changes moderately often, such as weekly.
Medium or larger sites with roughly 10,000 or more unique pages that change very rapidly, such as daily.
Sites with a large portion of URLs in Discovered – currently not indexed.

Google says these are rough estimates, not exact thresholds.

You probably do not need a crawl-budget project when:

Important pages are discovered and crawled within an acceptable time.
The site is small or moderately sized.
Host status is healthy.
The indexable inventory changes slowly.
A maintained sitemap and ordinary Page Indexing review are sufficient.

Do not create a complex crawl-budget dashboard simply because the term sounds important.

Crawl capacity limit

Google attempts to crawl without harming the server. Capacity reflects the maximum parallel connections and delay between fetches that Google's systems consider safe for a hostname.

Capacity can rise when:

Responses remain fast and stable.
Host connectivity and DNS are healthy.
The server handles requests without overload.

Capacity can fall when:

Response time rises.
5xx or 429 responses increase.
DNS, network, timeout, connection, or robots.txt availability failures occur.
Google detects that more crawling could overload the host.

Google also has finite crawling resources and makes choices across the web.

Capacity is hostname-specific. www.example.com and shop.example.com have separate crawl budgets in Google's model even when they belong to the same Domain property.

Crawl demand

Unused server capacity does not force Google to crawl more. Demand can remain low.

Google identifies several important demand factors:

Perceived inventory

Without clear controls, Google may try to crawl much of the URL inventory it discovers. Duplicate, removed, filtered, sorted, session, internal-search, and otherwise unwanted URLs can consume crawl attention.

Google calls perceived inventory the factor site owners can control most directly.

Popularity

URLs that are more popular on the web tend to be crawled more often so Google's systems can keep them current.

Staleness and update frequency

Google wants to refresh documents often enough to detect meaningful changes. A static policy page need not be crawled as frequently as fast-changing inventory or time-sensitive content.

Quality, relevance, and user value

Google says crawl demand for Search can account for overall user value, content uniqueness, page quality, and relevance. Faster servers alone do not create demand for interchangeable or low-value pages.

Site-wide events

Migrations and large launches can temporarily increase demand while Google reprocesses inventory.

Crawl budget is not an indexing budget

The processing sequence matters:

discover → schedule → crawl → render/process → consolidate → index decision

Crawl budget mainly affects scheduling and fetching. After a page is crawled, Google still evaluates:

Response and rendered content.
Indexing directives.
Duplicate and canonical clustering.
Content suitability and quality.
Other Search policies and systems.

If many URLs are Crawled – currently not indexed, increasing crawling is unlikely to be the primary solution. Investigate the pages and URL inventory.

Evidence of a real problem

Look for a consistent combination, not one metric:

Important new or updated pages wait unacceptably long for verified Googlebot crawls.
A large intended-canonical cohort remains Discovered – currently not indexed.
Googlebot spends a high share of requests on unwanted duplicate or parameter URLs.
Host status shows availability failures.
Response times or 5xx/429 rates correspond to crawl drops.
Canonical sitemaps contain far more valid URLs than Google discovers or crawls.
Server logs show stale important pages and repeated requests to low-value patterns.
A migration or large inventory change is not being processed within business requirements.

A daily fluctuation in Crawl Stats, a few unindexed pages, or a crawl of a tracking parameter is not sufficient evidence.

Build an intentional URL inventory

Classify every major URL pattern:

URL class	Index intent	Crawl intent	Control
Important canonical page	Index	Crawl	Internal links + sitemap + successful response
Duplicate with a retained user purpose	Consolidate	May crawl	Canonical signals
Retired URL with replacement	No	Crawl transition	Permanent redirect
Permanently removed URL	No	Decline over time	404 or 410
Filter/sort with no search value	No	Usually avoid	Prevent links, consolidate, or stable robots policy
Public page excluded from Search	No	Must crawl to see directive	`noindex`
Private content	No	No public crawl	Authentication
Infinite or broken generated path	No	No	Fix generation and links; truthful response or robots where appropriate

The policy must match the product experience. Do not block URLs that Google needs to render important pages.

Manage duplicate and unwanted inventory

Consolidate duplicates

Use redirects when retiring duplicate URLs and consistent canonical signals when variants must remain accessible. Update internal links and sitemaps to the preferred URLs.

Prevent crawl traps

Control:

Faceted navigation combinations.
Calendar paths.
Internal search URLs.
Session and tracking parameters in links.
Case and trailing-slash variants.
Infinite pagination.
Broken relative links.
Sort orders and view modes.

Fix link generation before relying on crawler directives.

Use robots.txt deliberately

Google's crawl-budget guide recommends robots.txt for pages or resources you do not want crawled at all. It also warns:

Do not use robots.txt as a temporary way to redistribute crawl budget.
Freed capacity will not necessarily move to other URLs unless Google was already hitting the site's serving limit.
A blocked URL can remain in the crawl queue and be retried when the block is removed.
Blocking does not guarantee that a URL will never be indexed from other evidence.

Do not block essential CSS, JavaScript, images, APIs, or pagination needed to understand important content.

Do not use `noindex` to save crawling

Google must crawl a page to see a noindex meta tag or HTTP header. The directive is correct for excluding a public page from Search, but it does not eliminate crawl cost.

Return 404 or 410 for permanent removals

Google says these are strong signals not to crawl a removed URL as frequently. Do not robots-block a removed URL before Google can process the truthful missing response.

Eliminate soft 404s

Empty or error-like pages returning 200 can continue to be crawled. Return a truthful status, restore useful content, or redirect to a close replacement.

Maintain accurate sitemaps

Canonical sitemaps help Google understand intended inventory.

Include only public, indexable canonical URLs.
Remove redirects, errors, noindex, blocked URLs, and duplicates.
Use accurate <lastmod> for significant updates.
Segment by stable URL cohorts for analysis.
Keep sitemap files reliably fetchable.

Do not repeatedly submit unchanged sitemaps or expect every listed URL to be crawled immediately.

Reduce redirect and fetch cost

Point internal links directly to final URLs.
Eliminate unnecessary redirect chains.
Avoid sitemap and canonical references to redirects.
Keep pages and required resources efficient to fetch and render.
Support conditional requests and 304 responses appropriately when content is unchanged.
Cache expensive but stable output safely.

Google says faster loading and rendering may allow it to read more content, but speed does not create demand for low-value content.

Protect and improve host capacity

Use Crawl Stats host status plus CDN, WAF, load balancer, origin, application, database, and dependency telemetry.

Track:

5xx and 429 response share.
DNS and connection failures.
robots.txt availability.
Response time by URL group and resource type.
Timeouts, resets, refused connections, and truncated responses.
Cache-hit rate and origin load.

Fix capacity constraints through caching, query optimization, resilient dependencies, resource scaling, and security-rule correction.

Do not use sustained 429 or 5xx as normal crawl management. They signal server trouble and can reduce crawling and eventually indexing.

Measure crawl efficiency with server logs

Crawl Stats is aggregated and shows sampled examples. Verified server and edge logs provide URL-level evidence.

Build internal metrics such as:

intended crawl share = verified Googlebot requests to intended URLs
                     / all verified Googlebot page requests

publication-to-crawl lag = first verified Googlebot request
                          - public publication time

stale refresh lag = latest meaningful page update
                  to next verified Googlebot refresh

These are your operational metrics, not Google-published scoring formulas.

Segment by:

Hostname.
URL pattern.
Response.
Crawl purpose.
Googlebot type.
Sitemap cohort.
Page importance and update frequency.

Verify Googlebot identity; user-agent strings can be spoofed.

Diagnose low crawling systematically

Confirm the site actually fits Google's crawl-budget audience.
Define important canonical URL cohorts and acceptable crawl latency.
Check Page Indexing for Discovered – currently not indexed.
Check Crawl Stats host status, responses, time, purpose, and crawler type.
Analyze verified logs for important and unwanted patterns.
Audit sitemap accuracy and internal discovery.
Find duplicate inventory, traps, redirects, soft 404s, and removed URLs.
Fix host capacity and serving failures.
Improve the usefulness and uniqueness of intended pages.
Measure cohort movement over weeks, not isolated daily volume.

Diagnose excessive crawling

Identify which crawler and URLs cause it:

New Search content can increase discovery crawling.
AdsBot can surge after Dynamic Search Ads target changes.
StoreBot can reflect shopping activity.
Page resource load can rise after frontend changes.
Crawler traps can generate effectively infinite URLs.

Protect server availability first, then fix the inventory or product-specific trigger. Google's Crawl Stats documentation provides short-term emergency measures for overload, but they should not replace capacity planning and URL control.

What Search Console can and cannot tell you

Search Console can show:

Aggregate crawl requests, size, and response time.
Host availability.
Response, file-type, purpose, and crawler-type patterns.
Example requested URLs.
Page Indexing cohorts and sitemap filters.
Indexed evidence for individual URLs.

It cannot show:

A single numeric crawl-budget allowance.
A complete URL-level crawl log.
Guaranteed next-crawl times.
Whether more crawling would cause indexing.
A direct “wasted crawl budget” metric.
Competitor crawl allocation.

Use logs and a classified URL inventory to answer the missing operational questions.

Common mistakes

Applying enterprise crawl-budget tactics to a small healthy site.
Treating more crawling as a ranking goal.
Trying to increase capacity when demand is the constraint.
Trying to improve demand while the host is failing.
Blocking URLs temporarily and assuming requests will transfer elsewhere.
Using noindex as a crawl-saving tool.
Blocking removed URLs instead of returning 404/410.
Including unwanted URLs in sitemaps and internal links.
Blocking resources required for rendering.
Measuring total requests without classifying URL patterns and crawler types.
Assuming every crawled page should be indexed.