Soft 404s Are Bleeding Your Crawl Budget

The page that lies about being alive

Most crawl problems announce themselves with an error status. A 404 says "not found." A 500 says "server error." A crawler reads the code, understands the situation, and acts accordingly. A soft 404 is dangerous precisely because it does the opposite: it returns 200 OK, the universal signal for "valid page, index me", while the content on the page is some variant of "nothing here." The HTTP layer says yes; the content says no; and the crawler initially believes the HTTP layer, then has to spend effort figuring out it was lied to.

Per Google's documentation on troubleshooting crawling errors, Google's systems inspect page content to decide whether a 200 response is a real page or a disguised error, and when content reads as "not found" or is too thin to be useful, Google classifies it as a soft 404. That classification is Google telling you it caught your page in a lie, and making it do that detection work is itself a cost, on top of the crawl it wasted fetching the page in the first place.

The 2025 confirmation: soft 404s do cost crawl budget

This used to be debated. The conventional wisdom was that error pages were cheap because Google learns to stop crawling them. But in 2025 Google explicitly confirmed the nuance that matters. Per Search Engine Journal's report on Google's statement, soft 404s consume crawl budget despite returning 200 OK, which is the opposite of how standard 4xx errors behave. The reason is structural: because the page returns 200, Google cannot dismiss it from the status code alone. It has to crawl the page, render or read the content, and run its soft-404 classification to determine the page is worthless. Every one of those steps is crawl capacity spent, and because the page keeps returning 200, Google keeps treating it as a legitimate URL worth revisiting.

That is the bleed. A true 404 teaches Google "this URL is dead, deprioritize it." A soft 404 teaches Google "this URL is alive, keep checking it", while the page never has any value. On a small site this rarely binds. On a large site, tens of thousands of URLs or more, it is a real constraint, and there is a documented case, per Search Engine Land's account of a soft-404-driven traffic collapse, of soft 404s and related indexing issues contributing to a roughly 90% traffic drop. The compounding effect is real and it slows discovery and indexation of the pages you actually want to rank.

Where soft 404s come from

Almost nobody builds a soft 404 deliberately. They emerge from how modern sites and frameworks handle absence.

The catch-all route that never 404s

Single-page apps and many frameworks route everything through one handler that returns 200 and renders a client-side view. When the route does not match real content, the app shows a "not found" component, but the server already returned 200 before the JavaScript decided the page was empty. Every nonexistent URL is now a soft 404, and SPAs can generate them by the thousand because any typo'd or stale URL resolves to a 200 shell.

Empty result and out-of-stock pages

A search-results page with zero results, a category page with no products, a discontinued product page that now shows "this item is no longer available", all commonly return 200 with effectively no useful content. Per Onely's explanation of soft 404s, e-commerce sites generate these by the thousand as inventory churns, and faceted navigation can multiply them across every empty filter combination.

Thin and placeholder pages

Auto-generated tag pages with one item, paginated archives past the last real page, or placeholder pages awaiting content all read as thin to Google and get classified as soft 404s even though you consider them legitimate. The threshold is Google's, not yours.

The friendly redirect to home

A well-meaning rule that redirects every unknown URL to the homepage with a 200 is a soft 404 generator, Google sees a "page" that is really just your homepage wearing a different URL, and flags the mismatch. This is a surprisingly common anti-pattern dressed up as good UX.

The crawl-budget bleed, concretely

Crawl budget is the finite attention Googlebot gives your site, how many pages it will fetch in a given window. Picture a site with 50,000 real URLs and 50,000 soft 404s from churned inventory and a catch-all route. Googlebot is now spending half its budget on pages that will never rank, will never convert, and exist only as the residue of how your framework handles absence. Your new product launches and fresh articles wait in line behind ten thousand "this item is no longer available" pages that keep returning 200 and keep inviting re-crawls.

Detect what a status check can't see

This is the detection trap: a normal uptime monitor or a status-code crawl sees 200 and reports the page healthy. The defect is the disagreement between status and content, so you have to evaluate both together, fetch the page, confirm it returns 200, and then check whether the content actually reads like a missing or empty page.

# Detect the lie: 200 status + "not found" / thin content
const SOFT_404_SIGNALS = [
  /page not found/i, /no longer available/i, /sorry, we couldn't find/i,
  /0 results/i, /no products? (found|match)/i, /this page (doesn't|does not) exist/i,
  /out of stock/i, /coming soon/i,
];

async function checkSoft404(url) {
  const res = await fetch(url);
  if (res.status !== 200) return null;          // honest error, not a soft 404
  const html = await res.text();
  const text = stripTags(html);

  const saysNotFound = SOFT_404_SIGNALS.some((re) => re.test(text));
  const isThin = text.trim().length < 200;       // almost no real content
  const noMainContent = !/<(article|main)[\s>]/i.test(html);

  if (saysNotFound || (isThin && noMainContent)) {
    return {url, status: 200, reason: saysNotFound ? 'not-found-text' : 'thin'};
  }
  return null;
}
// Output: URLs returning 200 that should be returning 404/410.

Google Search Console will surface soft 404s in its Pages (Index coverage) report under "Not indexed, " and you should watch it, but it reports them after Googlebot has already crawled, classified, and started excluding the pages. By the time it appears there, the crawl budget is already spent and the indexation delay is already happening. A proactive scan that reads intent-versus-status catches them before they enter the index, and before the bleed starts, by checking content and status together across your URL space.

Fix: make the status tell the truth

The fix is to align the HTTP status with reality. For a URL that genuinely has no content and never will, return a real 404 (not found) or 410 (gone, the stronger "this is permanently dead" signal that gets URLs dropped from the index faster). For a page that moved or has a clear successor, a discontinued product with a replacement, return a 301 to the relevant live page. For a page that should have content but is thin, the fix is content, not status codes: fill it, consolidate it into a richer page, or noindex it if it has a navigational purpose but no standalone value.

// SPA / framework: actually return 404 for unmatched routes,
// don't render a "not found" view behind a 200.
app.get('*', async (req, res) => {
  const data = await resolveContent(req.path);
  if (!data) {
    res.status(404);                 // tell the truth at the HTTP layer
    return res.send(renderNotFoundPage());  // helpful 404 UI is fine, just 404 it
  }
  res.status(200).send(renderPage(data));
});

// E-commerce: empty category / discontinued item
if (products.length === 0) {
  res.status(404);                   // or 301 to the parent category
}

The key insight for SPAs and frameworks: a helpful "page not found" screen is good UX, keep it, but it must be served with a 404 status, not a 200. The content and the status are two separate decisions, and the status is the one crawlers obey. For server-rendered apps this is straightforward; for client-rendered ones you may need server-side handling or prerendering so the bot gets the right status before any JavaScript runs.

Keep them from regrowing

Soft 404s regrow as inventory churns, content gets archived, and new routes get added without 404 handling. Make the intent-versus-status check continuous: crawl on a schedule, monitor the Search Console soft-404 report, and add a deploy-time assertion that key not-found paths actually return 404. Tighten the patterns the framework or CMS uses for empty states so a new empty category or zero-result page returns the right code from day one rather than becoming a soft 404 you discover months later in a coverage report, by which point it has been quietly eating crawl budget the whole time.

# Deploy-time assertion: known "should-404" paths must return 404
const SHOULD_404 = ['/this-page-does-not-exist-xyz', '/category/empty-test'];
for (const path of SHOULD_404) {
  const res = await fetch(SITE + path);
  if (res.status === 200) {
    console.error(`SOFT 404: ${path} returns 200, expected 404`);
    process.exit(1);
  }
}

The bottom line

A soft 404 is a page that lies to crawlers, content that says "not found" behind a 200 status that says "index me", and Google confirmed in 2025 that the lie costs you crawl budget twice: the bot keeps re-fetching a worthless page, and spends algorithmic effort detecting the mismatch, neither of which happens with an honest 404. On a large site, that bleed slows indexation of the pages you actually care about and has been linked to severe traffic collapses. The defect is invisible to any check that only reads status codes, because the whole problem is that the status code is wrong; you have to evaluate intent against status together. Return honest 404s and 410s for dead URLs, 301s for moved ones, real content for thin ones, scan continuously for the 200s that should not be, and assert it at deploy time. Honest status codes are free; soft 404s are a tax you pay on every crawl.