The Crawl Stats report shows Google's crawl requests to a site over time: request volume, bytes downloaded, average response time, host availability, response outcomes, file types, crawl purposes, and Google crawler types.
Use it to diagnose site-level serving and crawl-pattern changes. It is not a rank report, a complete server log, a URL-level crawl-history search, or a measure of how many crawled pages were indexed.
Availability and intended audience
Open Settings → Crawl stats.
The report is available only for root-level properties:
- A Domain property such as
example.com. - A root URL-prefix property such as
https://example.com/.
A path property such as https://example.com/store/ does not receive its own Crawl Stats report.
Google describes this as an advanced report. Sites with fewer than about 1,000 pages generally do not need this level of crawl analysis unless they are debugging an availability incident.
What the top metrics mean
Total crawl requests
The number of requests Google issued for site URLs and same-site resources, whether successful or not. Repeated requests for the same URL count separately.
The total can include:
- HTML pages.
- Images, JavaScript, CSS, video, PDFs, feeds, and other files.
- Each hop of a server-side redirect chain.
- Failed DNS or connectivity requests.
- Potential requests abandoned because robots.txt was insufficiently available.
It is a request count, not a unique-URL count.
Total download size
Bytes Google downloaded during crawling. A cached resource used by multiple pages may be downloaded only when needed rather than for every page render.
Use the metric to detect bandwidth changes, not to infer page count. One large video or script can outweigh many HTML responses.
Average response time
The average time for the site to respond to Google's crawl requests across fetched resources. Each resource is a separate observation.
This is not:
- Core Web Vitals.
- Browser rendering time.
- Largest Contentful Paint.
- A user-experience score.
- A pure HTML time-to-first-byte metric.
Segment by file type and date before attributing a rise to page templates.
Understand the report's URL accounting
Unlike Performance data, Crawl Stats uses the actual URL Google requested; it does not assign requests to canonical URLs.
Important consequences:
- HTTP and HTTPS requests can both appear.
- Duplicate and parameter URLs appear as requested.
- Every redirect hop is counted.
- Domain properties can include child hosts.
- A parent view can aggregate subdomains, while the Hosts list can expose active child hosts.
For URL-prefix properties, crawl data can still include both HTTP and HTTPS protocols, although example URLs are limited to the property's protocol.
If a page on this site loads a resource from another domain, that external request does not count here. If an external page loads a resource hosted on this property, the property's request can appear without context explaining the external parent page.
Host status
Host status summarizes significant availability problems during the previous 90 days.
The high-level states are:
- No significant availability issues. No major crawl availability problem detected in the period.
- Some availability issues, but not recently. A significant issue occurred more than a week ago.
- Recent availability issue. A significant issue occurred within the last week.
Expand host details for:
- robots.txt availability.
- DNS resolution.
- Host connectivity.
A green host status does not mean every request succeeded. It means Google found no significant host-level availability issue under its thresholds. Review response trends for template-specific or intermittent failures.
robots.txt availability behavior
Google does not require a site to publish robots.txt. These responses are accepted for the file:
- 200 with a file, including an empty file.
- 403, 404, or 410, interpreted as no robots.txt file.
429 and 5xx are unsuccessful responses.
Google's current Crawl Stats documentation describes this process:
- Google uses a recent successful robots response less than 24 hours old.
- If it needs a fresh copy and the request fails, crawling initially stops while Google retries.
- For the first 12 hours, Google stops crawling the site and keeps requesting robots.txt.
- From 12 hours to 30 days, Google can use the last successfully fetched version while continuing retries.
- After 30 days, behavior depends on whether the homepage is available: Google may act as if no robots file exists, or stop crawling when the homepage is unavailable.
Potential crawls abandoned during robots unavailability can count in total requests even though no page request reached the server. This can make Search Console totals much higher than origin-log counts.
Keep robots.txt fast, stable, and monitored separately.
Crawl responses
This table groups each request outcome as a percentage of responses, not unique URLs.
If Google requests one URL twice and receives 500 once and 200 once, the table reflects one success and one server error.
Normally healthy responses
- 200 OK.
- 301/308 permanent redirects when intentionally moved.
- 302/307 temporary redirects when truly temporary.
- 304 Not Modified.
Context-dependent response
- 404 can be correct for genuinely removed or nonexistent URLs. Investigate internal links and sitemaps; do not try to eliminate every external typo.
Responses to investigate
- robots.txt not available.
- Unauthorized 401/407 on intended public pages.
- Server error 5xx.
- Other 4xx.
- DNS unresponsive or other DNS error.
- Fetch, connection, timeout, unreachable, or redirect errors.
Click a response group to see its trend, request details, and example URLs.
Crawled file types
The report can group requests as:
- HTML.
- Image.
- Video.
- JavaScript.
- CSS.
- PDF.
- XML.
- JSON.
- Syndication feeds.
- Audio.
- Geographic data.
- Other or unknown types.
Use this view to explain bandwidth and latency changes. Examples:
- A frontend release causes far more JavaScript and CSS resource fetches.
- An image migration creates a surge in image requests.
- A slow API or JSON route increases average response time.
- Google spends requests on generated feeds or assets that should not be exposed.
Do not block essential rendering resources solely because their volume looks high. Google may need CSS, JavaScript, images, and APIs to understand pages.
Crawl purpose
Google groups purpose into:
- Discovery: The URL had not previously been crawled by Google.
- Refresh: A recrawl of a known URL.
After launching a large section or submitting a sitemap, an increase in discovery crawling can be expected. A mature site with relatively stable pages may show mostly refresh crawling.
Purpose does not tell you whether the request resulted in indexing.
Googlebot type
Common groups include:
- Smartphone.
- Desktop.
- Image.
- Video.
- Page resource load.
- AdsBot.
- StoreBot.
- Other Google agents.
Page resource load represents secondary fetches needed to render pages, such as CSS or images. An image fetched as a page resource can therefore appear under Page resource load rather than Googlebot Image.
Crawler-type segmentation is essential when diagnosing a spike. AdsBot or StoreBot behavior has different triggers from Google Search's primary page crawling.
Example URLs are samples
Clicking a group shows example requests with URL, time, response, and size information. The examples are representative, not comprehensive. Their absence does not prove Google did not request a URL.
Search Console says example weighting can vary by day. Do not export examples and call them the complete crawl log.
Use server, CDN, and WAF logs for URL-level history and verification.
Investigate a crawl spike
Start by asking whether the spike is expected:
- Large content launch.
- Previously blocked section opened.
- Site migration.
- New sitemap or many accurate
<lastmod>updates. - Image, video, or asset migration.
- Dynamic Search Ads targets.
- Store/product changes.
- URL explosion from filters, sessions, calendars, or broken links.
Then segment by:
- Host.
- Googlebot type.
- Crawl purpose.
- File type.
- Response code.
- Download size and response time.
The sequence distinguishes valuable new-page discovery from an AdsBot surge, resource refetch, redirect chain, or crawler trap.
If crawling causes real availability problems, use infrastructure protection and Google's documented emergency controls carefully. Short-lived 503 or 429 responses can signal overload, but Google warns that returning them for more than two or three days can reduce long-term crawling. Do not use prolonged server errors as routine crawl management.
Investigate a crawl drop
Compare the drop with:
- A new or broader robots.txt rule.
- Increased response time.
- More 5xx, 429, DNS, or network errors.
- Reduced inventory after a migration or cleanup.
- Lower update frequency.
- Changes to internal links or sitemaps.
- A decline in discovery demand.
- A page-quality or site-value problem.
Google says crawl rates are often relatively stable across a week or two, so focus on sustained and meaningful changes rather than daily noise.
You cannot command Google to increase crawl rate. For a small or medium site, update accurate sitemaps and verify that important pages and resources are not blocked. For a large site, analyze capacity, demand, and URL inventory.
Investigate slow response time
- Find the first sustained change date.
- Segment by host and file type.
- Compare request volume with latency; overload may be volume-driven.
- Review 5xx, 429, timeout, and connection trends.
- Correlate with deployments, cache changes, database load, third-party dependencies, and CDN behavior.
- Use example URLs to identify templates, then confirm with logs and application tracing.
Fix systemic causes:
- Expensive dynamic generation.
- Cache misses or invalidation storms.
- Slow database queries.
- Large resource payloads.
- Redirect chains.
- Failing upstream services.
- Edge-to-origin routing problems.
Do not use Crawl Stats response time to claim a Core Web Vitals improvement.
Compare Crawl Stats with server logs
Differences are normal because:
- Search Console reports most but not necessarily all crawl requests.
- Examples are sampled.
- Requests can fail before reaching your origin.
- CDN-served responses may not appear in origin logs.
- robots.txt-unavailable potential requests can count without being made.
- Bot user-agent filters in logs may include spoofed traffic.
- Time zones and date windows may differ.
- Search Console includes multiple Google crawler types and resources.
For a reliable comparison:
- Use the same timezone and window.
- Include edge and origin logs.
- Verify Google crawler identity.
- Count requests, not unique URLs.
- Separate crawler type, response, host, and file type.
- Account for every redirect hop.
What Crawl Stats cannot answer
It cannot directly tell you:
- Whether a specific URL is indexed now.
- Why a crawled page was not indexed.
- A complete list of requested URLs.
- The crawl budget Google assigned as one numeric allowance.
- Whether Google “likes” a page.
- Rankings, clicks, impressions, or conversions.
- Core Web Vitals or user page speed.
- Which external page caused a hosted resource to be fetched.
Use URL Inspection, Page Indexing, Performance, field-speed tools, server logs, and analytics for those questions.
Monthly review checklist
- Host status has no recent significant issue.
- robots.txt availability is stable.
- DNS and host connectivity are healthy.
- 5xx, 429, timeouts, and other bad responses are controlled.
- Average response time has no unexplained sustained rise.
- Crawl volume changes match releases, migrations, or inventory changes.
- Discovery and refresh proportions make sense.
- Googlebot-type changes have an identified cause.
- File-type bandwidth matches intended resources.
- 404 and redirect volume is expected after lifecycle changes.
- Server logs confirm important URLs are crawled and expose unwanted patterns.
Common mistakes
- Treating requests as unique pages.
- Treating crawled URLs as indexed URLs.
- Comparing average response time with Core Web Vitals.
- Assuming examples are exhaustive.
- Ignoring crawler type during a spike.
- Looking only at origin logs when a CDN serves Google.
- Trying to fix every 404.
- Blocking all resources to reduce request volume.
- Interpreting a lower crawl rate as automatically bad.
- Trying to force a higher rate before improving inventory, capacity, and content usefulness.