Troubleshoot “Couldn't fetch”

Diagnose sitemap access, response, robots, redirect, and processing problems systematically.

Directional evidence Partially automatable Intermediate

Couldn't fetch in Search Console's Sitemaps report means Google could not retrieve the sitemap file for the report's latest request. It is a transport, access, availability, property, or crawl-demand problem—not the same as a sitemap that was fetched but contains XML or URL errors.

Open the sitemap's details before changing the file. The details and timestamps identify which request and error you are investigating.

Distinguish fetch failure from parsing failure

Report result	What happened	First investigation
Success	File was fetched and processed without reported errors	Analyze discovered URLs and Page Indexing outcomes
Couldn't fetch	Google could not retrieve the sitemap file	Access, response, robots, property, server, manual action, or crawl demand
Sitemap had X errors	File was retrieved and at least partly read	XML syntax, encoding, URL scope, redirects, tags, or individual URLs

Rewriting valid XML cannot fix a firewall that blocks Google. Changing CDN rules cannot fix malformed XML after a successful fetch. Start in the correct layer.

Officially documented fetch causes

Google currently lists these broad reasons:

The sitemap is blocked by robots.txt.
The site has an unresolved manual action; Google says sitemaps are not read in that condition.
The submitted sitemap URL is wrong or returns 404.
A general availability problem prevents retrieval, such as server failure.
Crawl demand for the sitemap is low.

Operationally, the general failure category can involve authentication, 403, 429, 5xx, DNS, TLS, connection, timeout, redirect, CDN, WAF, or malformed response behavior. Use the details and live test rather than guessing.

Step 1: confirm the exact submitted URL and property

Copy the sitemap URL from the Sitemaps report. Do not reconstruct it from memory.

Check:

http versus https.
www versus non-www.
Hostname and port.
Path, case, filename, and extension.
URL-prefix property scope.
Whether a migration changed the location.
Whether a submitted sitemap index points to the expected child files.

The Sitemaps report shows submissions for the current property. A sitemap expected in another protocol or hostname property will not necessarily appear here.

Request the exact URL in a signed-out browser or HTTP client. It should be public without cookies, login, geolocation, or prior navigation.

Step 2: run a live URL Inspection

Google's Sitemaps help specifically recommends this workflow:

Copy the sitemap URL from its details page.
Paste it into URL Inspection.
Click Test live URL.
Expand page availability.
Confirm Crawl allowed? Yes.
Confirm Page fetch: Successful.

The test can reveal a robots block, access response, server error, or other current fetch failure.

A successful live test after a historical failure can mean:

The failure was transient.
The issue was fixed after Google's last sitemap request.
An intermittent or regional condition did not occur during the test.

Correlate test time with the report's last-read attempt and infrastructure logs.

Step 3: check robots.txt

Google respects robots.txt when fetching sitemaps. A Sitemap: declaration does not override a Disallow rule that blocks the sitemap URL.

Example accidental block:

User-agent: Googlebot
Disallow: /generated/

Sitemap: https://example.com/generated/sitemap.xml

If the sitemap lives under /generated/, the rule prevents Googlebot from fetching it.

Use the robots.txt report and live inspection to identify the matching rule. Place sitemaps at a stable public path and avoid broad rules that cover them.

Do not block a sitemap merely because some listed pages should not be crawled. Clean those URLs out of the sitemap and manage their page-level crawl policy separately.

Step 4: inspect the actual HTTP transaction

Check the status, redirects, headers, and response body:

curl --head https://example.com/sitemap.xml
curl --location --output /tmp/sitemap.xml \
  --write-out '%{http_code} %{url_effective}\n' \
  https://example.com/sitemap.xml

Verify:

A stable successful response.
The final destination is the intended sitemap, not a homepage, login, challenge, or error template.
Redirects are finite, valid, and consistent.
The body is complete and not truncated.
Compression is valid if using .gz.
Content is the expected XML, text, RSS, or Atom format.
The response works without cookies or browser-only headers.

A browser displaying XML is useful but not conclusive. Browsers can carry login sessions, bypass cached challenges, or use a different network path.

Step 5: review CDN, WAF, and bot controls

Security systems often distinguish requests by IP, user agent, geography, reputation, request rate, or missing browser features.

Look for:

403 challenges.
429 rate limits.
5xx generated at the edge.
JavaScript or CAPTCHA challenges.
Geographic restrictions.
Hotlink or header requirements.
Rules blocking .xml or large files.
Cached error responses.

Use CDN and firewall logs for the report's request window. Verify crawler identity through Google's documented methods before allowlisting IPs; a Googlebot user-agent string can be spoofed.

The sitemap should not require JavaScript execution or a browser challenge.

Step 6: check host, DNS, TLS, and network health

Inspect Search Console Crawl Stats → Host status for DNS resolution and host connectivity problems. Also review:

DNS records and name servers.
IPv4 and IPv6 reachability when both are published.
TLS certificate validity and hostname coverage.
CDN-to-origin health.
Connection resets and refused connections.
Response and upstream timeouts.
Deployment or maintenance incidents.
Large dynamic sitemap generation that exceeds platform time limits.

Generate large sitemaps ahead of time or cache them rather than executing an expensive database export on every request.

Step 7: check the Manual Actions report

Google's Sitemaps documentation says a site with an unresolved manual action can have sitemaps left unread. Open Security & Manual Actions → Manual actions.

If an action exists:

Understand its scope.
Correct the underlying violation across the site.
Submit a reconsideration request with accurate evidence when appropriate.
Resubmit the sitemap after the action is resolved.

Do not treat sitemap resubmission as a workaround for the action.

Step 8: validate the file after fetch succeeds

Once the file is retrievable, check the next layer:

UTF-8 encoding.
Valid XML and namespace.
Correct sitemap or sitemap-index root element.
Absolute and properly escaped URLs.
50,000 URL and 50 MB uncompressed limits.
Valid gzip when compressed.
Child sitemap directory and site scope.
Correct host and protocol for listed URLs.

A fetch success can then become Sitemap had X errors if the content is malformed. That is progress because Google reached the file, but parsing still needs repair.

Sitemap-index and child-sitemap failures

Test every level separately:

sitemap index
  ├── products-1.xml
  ├── products-2.xml
  └── categories.xml

The index can be fetched while one child fails. A child can also be valid but outside the index's permitted directory or site scope.

For each child:

Request it directly.
Confirm public access and response stability.
Validate its format and limits.
Check its own status in Search Console details.
Confirm the index <loc> contains an absolute, escaped URL.

Do not assume top-level success proves every child was processed.

Why a success in the past still matters

Google says that after it successfully reads a sitemap, it remembers that information. A later fetch failure does not make Google forget every previously discovered URL; newer information can overwrite older information for a URL.

However, persistent failure prevents new URLs and accurate updates from being processed. It also removes a valuable diagnostic channel.

Fix the file even when older pages remain indexed.

Retry and resubmit behavior

After a fetch or critical parse failure, Google retries for a few days. If failure continues, it eventually stops trying to read that sitemap URL.

Use this sequence:

Identify the latest reported cause.
Fix the underlying access, availability, or file problem.
Confirm multiple successful public requests.
Run live URL Inspection.
Resubmit the sitemap once in Search Console.
Monitor the last-read date and status.

Do not repeatedly resubmit an unchanged, inaccessible file. Resubmission does not repair the server or increase crawl demand by itself.

If everything tests successfully but status remains stale

Check chronology first:

Did the report's failed request occur before the fix?
Has a new last-read attempt happened?
Does the sitemap row show the correct property and URL?
Do verified Google requests appear in edge or server logs?
Is the failure intermittent?

Allow time for a new request. If Google has stopped retrying after prolonged failure, resubmit once after confirming the fix.

Do not conclude the report is broken solely because your browser can fetch the file now.

Prevent future fetch failures

Host the sitemap at a stable public URL.
Generate and cache it reliably.
Monitor status, latency, size, and response body.
Alert on redirects, 4xx, 429, 5xx, truncation, invalid gzip, and unexpected HTML.
Keep it outside authenticated and bot-challenged paths.
Test robots rules on every deployment.
Validate sitemap indexes and children.
Record last successful generation and URL count.
Keep an older valid file available during generation rather than serving a partial file.
Use atomic publication: generate a complete file, validate it, then replace the public version.

Common mistakes

Debugging XML before confirming Google can fetch the file.
Testing a different URL from the submitted one.
Using a logged-in browser as the only availability test.
Declaring the sitemap in robots.txt while also blocking its path.
Serving a CAPTCHA, JavaScript challenge, or HTML login page.
Dynamically generating a large sitemap until the request times out.
Ignoring child files because the index itself succeeds.
Resubmitting repeatedly without a fix.
Moving the file without updating Search Console and robots.txt.
Assuming previously discovered URLs vanish after one failed fetch.
Treating a successful fetch as proof that listed pages are indexed.