Indexing HTTP errors

Respond correctly to 401, 403, 404, soft 404, other 4xx, and 5xx indexing statuses.

Partially automatable Intermediate

HTTP responses tell Google whether a URL returned usable content, moved, does not exist, requires access, or failed at the server. The Page Indexing report turns several response outcomes into reasons such as Server error (5xx), Redirect error, Blocked due to unauthorized request (401), Not found (404), Blocked due to access forbidden (403), other 4xx, and Soft 404.

The correct goal is not “make every URL return 200.” Each URL should return the response that truthfully represents its state.

Response decision table

URL state	Appropriate outcome	Search consequence
Valid public page	`200 OK` with meaningful content	Eligible for indexing, not guaranteed to be indexed
Permanently moved page	Permanent redirect to a close replacement	Redirecting URL is normally not indexed; target is evaluated separately
Temporarily moved page	Temporary redirect where semantically correct	Google follows the destination but treats the signal as temporary/weak
Permanently removed with no replacement	`404 Not Found` or `410 Gone`	URL is not indexed and an indexed URL is removed over time
Private or account-only content	Authentication such as `401`, login, or access control	Content remains unavailable to Search, as intended
Public page accidentally denied	Fix `401`/`403`, WAF, CDN, or application rule	Page can be fetched after access is restored
Application or infrastructure failure	Fix `5xx`, timeout, reset, DNS, or connection failure	Repeated failures reduce crawling and can eventually remove indexed URLs
Overloaded server	Fix capacity causing `429` or `5xx`	Google treats 429 as a server error and slows crawling
Missing content returned as 200	Return 404/410, redirect appropriately, or restore useful content	Prevents a soft 404 classification

How Google handles response classes

2xx success

A successful response lets Google pass the content into further processing. It does not guarantee indexing. If a 2xx page contains an error message, empty state, or “not found” experience, Google may classify it as a soft 404.

204 No Content provides no body for processing and is not a substitute for a valid indexable page.

3xx redirects

Google processes the final destination rather than the content of the redirecting URL. Permanent redirects such as 301 and 308 are strong signals that the destination should be canonical. Temporary redirects such as 302 and 307 are weaker signals.

A valid redirect normally appears as Page with redirect. A broken chain, loop, malformed or empty target, excessive chain, or eventual overlong URL can become Redirect error.

4xx client errors

Google does not use the response body from 4xx URLs for indexing. For Google Search, new 4xx URLs are not indexed and previously indexed URLs are removed over time.

Google currently handles ordinary 4xx responses, including 404 and 410, in the same general way for Search. The exception is 429 Too Many Requests, which Google treats as a server-overload error.

Ordinary 4xx responses do not reduce the site's crawl rate, although Google gradually requests a persistently missing URL less often. Do not use 401 or 403 to try to manage crawl rate.

5xx server errors and 429

Google ignores content returned with a 5xx response. Repeated 5xx and 429 errors cause its crawlers to slow down. Already indexed URLs can be preserved temporarily but are eventually dropped if the failures persist.

When the server returns successful responses again, Google gradually increases crawling. Recovery is not necessarily immediate.

Server error (5xx)

This Page Indexing reason means Google could not get a valid response because the server returned a 500-level error, timed out, was busy, or encountered a related connectivity failure.

Common underlying sources include:

Application exceptions.
Database, cache, queue, storage, or upstream API failures.
Origin overload or resource exhaustion.
CDN, reverse-proxy, or load-balancer errors.
Serverless or application timeouts.
Deployment restarts and maintenance windows.
WAF or bot-management rules returning a server response.
TLS, DNS, connection, and network failures.
Pages whose dynamic generation is much slower for certain parameters.

Diagnose historical and current evidence

Open the affected reason in Page Indexing and note the trend and example URLs.
Inspect representative URLs and record Google's last crawl date and fetch result.
Run Test live URL to see whether the failure is reproducible now.
Open Settings → Crawl stats and inspect Host status, response codes, response time, and failure dates.
Correlate the exact period with CDN, firewall, load balancer, origin, application, database, and dependency logs.
Group failures by hostname, route, template, status code, origin, region, and deployment.

A successful live test does not prove the historical failure was false. Transient errors, peak load, regional routing, and bot-specific security rules may be absent during the test.

Fix the system, not only example URLs

Restore stable successful responses for public pages.
Correct the exception, dependency, query, or timeout causing failures.
Cache expensive responses where safe.
Reduce unnecessary dynamic URL combinations.
Scale or tune infrastructure if legitimate traffic and crawling exceed capacity.
Configure health checks and load balancing correctly.
Stop CDNs or WAFs from misclassifying verified Google crawlers.
Keep error pages lightweight and preserve the correct 5xx status during real failures.

Do not change a genuine server failure to 200 OK while still serving the error message. That hides monitoring evidence and can produce a soft 404.

Network and DNS errors

Some failures occur before the application can return an HTTP status. Google treats network timeouts, connection resets, and DNS failures similarly to 5xx errors: crawling slows because the host appears unable to serve requests.

Review:

DNS A, AAAA, CNAME, and name-server records.
Recent DNS changes and propagation.
TLS certificate validity and hostname coverage.
Firewall and network access-control logs.
Connection resets, refused connections, truncated responses, and timeouts.
CDN-to-origin routing and health.
Whether both IPv4 and IPv6 routes work when published.

Search Console requests that never reached your server will not appear in origin logs. Combine Crawl Stats with DNS, CDN, edge, and network telemetry.

robots.txt availability is host-critical

Before crawling pages, Google needs to know whether robots.txt permits the request. The site does not have to publish a robots.txt file: a 404, 403, or 410 response for /robots.txt is treated as no robots file and can be acceptable.

By contrast, repeated 429, 5xx, DNS, or connection failures for robots.txt can postpone or halt crawling because Google cannot safely determine the rules. Search Console's Crawl Stats documentation says that if robots.txt remains unavailable for a day, Google can stop crawling temporarily until an acceptable response returns.

Monitor /robots.txt separately from ordinary application pages and keep it fast and stable.

Blocked due to unauthorized request (401)

HTTP 401 means the resource requires authentication.

Leave it protected when

It contains private, paid, account-specific, administrative, or confidential information.
It is not intended as a public search landing page.

Fix it when

The public page accidentally inherited an authentication rule.
A deployment, proxy, staging gate, or application middleware protected the wrong route.

Test in a signed-out private session. If the private content needs a public search presence, create a separate public landing page rather than exposing the protected resource.

Blocked due to access forbidden (403)

HTTP 403 means the request was understood but denied. For an intended public page, common causes are:

CDN bot protection.
WAF rules.
IP, ASN, region, or rate restrictions.
User-agent rules.
File permissions.
Application authorization.
Hotlink or request-header requirements.

Googlebot does not provide login credentials. If the page is meant to be public, allow ordinary signed-out access or adjust the blocking rule. When making crawler-specific firewall decisions, verify Google's crawler identity through Google's documented reverse- and forward-DNS or published-IP methods; user-agent strings are easy to spoof.

Do not bypass authentication for confidential content simply because a request claims to be Googlebot.

URL blocked due to other 4xx issue

Identify the exact status and why the route returns it. Examples can include invalid request handling, unsupported methods, legal restrictions, or custom gateway behavior.

Ask:

Should the URL exist?
Should it be public?
Is the response produced by the origin, CDN, WAF, or application?
Does the status vary by method, user agent, IP, region, cookie, or header?
Is an internal link or sitemap generating malformed URLs?

Return a status that matches the intended state. Do not convert all 4xx responses to 200 merely to clear the report.

Not found (404) and gone (410)

A 404 is not inherently an SEO problem. Google says most 404s are not worth fixing when the URL genuinely should not exist.

Keep the 404 or 410 when

Content was permanently removed and has no close replacement.
The URL never existed and has no meaningful links or demand.
A malformed external URL is outside your control.

Redirect when

The content moved to a close, useful replacement.
A common misspelling or old route has a clear equivalent.
A migration changed the URL while preserving the page's purpose.

Restore the page when

It was removed accidentally.
It remains a valid, valuable destination.

Clean the discovery source

If a 404 appears in a sitemap, remove or correct it. If internal links point to it, update them. Google can continue retrying known missing URLs for a while, but the frequency declines.

Do not redirect every removed URL to the homepage or an unrelated category. That is poor for users and can be treated as a soft 404.

Soft 404

A soft 404 occurs when a URL returns a success-like response but Google's systems judge the content to represent an error or missing page.

Common examples:

A custom “page not found” template returning 200.
An empty product, category, profile, or search page.
An error or access message inside a normal site template.
A redirect to an irrelevant destination.
A blank page after rendering fails.

Choose the truthful resolution:

Real missing content with no replacement → 404 or 410.
Moved content → permanent redirect to the closest equivalent.
Valid page → successful response with meaningful, renderable content.

Use live URL Inspection and View tested page to see the HTML and screenshot Google receives.

Redirect error

Trace the full chain from the reported URL:

requested URL
  → hop 1
  → hop 2
  → final URL and response

Look for:

Loops between two or more URLs.
Chains created by protocol, hostname, locale, slash, and migration rules stacking.
Relative or malformed Location headers.
Empty destinations.
Encoded parameters that grow on every hop.
A final URL that errors, blocks access, or redirects again based on cookies or region.

Make each old URL resolve directly to one final, public, canonical destination where practical. Update internal links, sitemaps, canonicals, hreflang, and structured data to use the final URL.

Test responses safely

An HTTP client can expose the current chain and headers:

curl --head https://example.com/page
curl --location --output /dev/null \
  --write-out '%{http_code} %{url_effective}\n' \
  https://example.com/old-page

Also test through:

A signed-out browser.
URL Inspection's live test.
Multiple relevant regions or network paths when the service is geo-distributed.
The same CDN and hostname users reach.

Do not rely only on a desktop browser status. Crawling failures can be intermittent and may originate at an edge or firewall before reaching the application.

Monitor response health at scale

Track at minimum:

Request timestamp.
Hostname and URL pattern.
Status code.
Response time.
Edge and origin identifiers.
User agent and verified crawler status when relevant.
Cache status.
Deployment or incident identifier.
Redirect destination.

Alert on:

A sustained rise in 5xx or 429.
robots.txt availability failures.
401/403 appearing on previously public templates.
Redirect loops or increasing chain length.
Success responses whose content is an error state.
A falling indexed count corresponding to response failures.

Crawl Stats aggregates requests and responses rather than listing a complete URL-level crawl history. Server and edge logs provide the detailed evidence.

Validate fixes in Search Console

After deploying a systemic fix:

Test representative URLs from every affected template.
Confirm the current response and rendered result.
Verify Crawl Stats host status is healthy.
Start Validate fix for the Page Indexing reason when all known affected instances should be corrected.
Monitor the validation log and new crawl data.

Google says validation often takes up to about two weeks and can take longer. A remaining bad example can stop an attempt. Google also detects fixes during normal crawling without validation.

Do not validate correct 404s, authentication, redirects, or other intentional outcomes just to eliminate a row.

Common mistakes

Returning 200 for every route.
Redirecting every removed URL to the homepage.
Assuming one successful live test disproves an intermittent failure.
Using 401, 403, 429, or 5xx as a crawl-control strategy.
Allowlisting spoofed Googlebot user agents without verifying origin.
Exposing private content to make an indexing status green.
Ignoring CDN, firewall, DNS, and robots.txt because the origin looks healthy.
Leaving redirecting, erroring, or removed URLs in canonical sitemaps.
Fixing report examples but not the template, router, or infrastructure rule.
Expecting a recovered server to regain its previous crawl rate instantly.