Index bloat: when the fix is deleting pages

What is index bloat?

More URLs eligible for indexing than the site has distinct things to say. It shows up as pagination, filter parameters, tag archives, per-answer or per-comment permalinks, and templated pages generated for combinations nobody searches for. Each one returns 200 and looks healthy in a crawl report. Nothing is broken. That is the difficulty: a crawler grades pages one at a time and index bloat is a property of the set.

The remedy is unpopular because it means deleting work someone was paid to produce. It is also, at scale, one of the few interventions with a documented before and after.

The version where deleting 5 million pages was correct

Grant Simmons was hired into Homes.com in 2014 after the site lost roughly 60% of its organic traffic to what turned out to be Panda. The diagnosis was not subtle:

We should clean up a lot of crap, a lot of thin pages that existed purely for just existing

The scale of the cut is the part people do not believe:

it was a matter of getting rid of millions of pages from a 110 million page site, taking some really big risks in chopping.

Five million pages came out of the index. Grant then did the thing every SEO fantasises about, which is get a straight answer from Google in person. At SMX London, with John Mueller in the audience, he described the site and the remediation:

He got on his laptop, he typed in and 20 seconds later he goes, yeah, you're under a Panda penalty. And I go, all right, well, they know that stuff.

And the outcome:

we made those changes and when Panda was pushed out at that time, the rolling updates, we got back all our traffic and then some.

His summary of why the incumbent agency had not fixed it is the transferable lesson:

They were not thinking to be aggressive. I've seen in those days recovery had to be aggressive.

Grant's full episode covers entity SEO, measurement, and what he thinks reporting still gets wrong: listen here.

Three things made deletion the right call at Homes.com and they are worth stating, because they are the conditions most sites fail to meet. The pages were near-duplicates generated by a template. There were millions of them, enough to change how the whole domain was assessed. And there was a measured traffic collapse to recover, not a hunch.

The version where deleting is a waste of a weekend

Now the small-site case, from a crawl of seoarcade.com.

The site runs a Q&A section. Every question has a page, and every individual answer to that question also gets its own permalink. Filtering the crawl to a single question about keyword forecasts returns 20 of the 867 crawled URLs. Eight of those came back 200, all carrying the same title and between 824 and 1,127 words of overlapping text.

Each of those eight sits in the sitemap and reports isIndexable: true. By any crawl-based definition this is index bloat, and the obvious move is to noindex the answer permalinks and ship it.

Then you check what Google did with them, which is the step almost nobody takes.

Search Console URL inspection on six URLs from that site returned "URL is unknown to Google" for five of them: the keyword-forecasts question page, two of its answer permalinks, a third answer permalink from a different question, and the parameter archive at /?post_type=answer. The only one that came back "Submitted and indexed" was an ordinary question page, which is the page that should be indexed. It reported a Google-selected canonical matching its declared canonical, three referring URLs, and a Q&A rich result.

There was nothing to remove. The crawler found a mess and the index did not have one.

Check before you cut

This is the difference between the two cases, and it costs nothing to establish. Google's URL Inspection API costs nothing and uses no OpenSEO credits, and it answers the only question that matters: does Google know this URL exists, and if it does, which URL did it pick as canonical?

Four verdicts come back, and each implies a different action.

• If the URL is unknown to Google, there is nothing to remove. Fix the sitemap so you stop submitting it, and move on.
• "Duplicate, Google chose a different canonical" also leaves you nothing to remove, because Google has already consolidated the set. Confirm the URL it picked is the one you want.
• The real bloat signal is "Crawled, currently not indexed" or "Discovered, currently not indexed". Google has seen the page and declined it, so either make the page worth indexing or take it out of the sitemap and the internal link graph.
• A page that comes back "Submitted and indexed" when you never wanted it indexed is the one case for noindex or removal.

Patrick Stox, who built Site Audit and Page Inspect at Ahrefs, points out that the discovery side has more inputs than most people account for, which is why bloat keeps reappearing after you delete it:

But what are the data sources for these systems? Most people will tell you sitemaps, or crawling. Well, there's more. RSS feeds are a thing.

Removing a URL from a sitemap does not remove it from an RSS feed, an internal link, or somebody else's page. If a URL keeps coming back after you deleted it, one of the other sources is still publishing it. Patrick's episode on building for the GEO era is on the Unscripted SEO podcast.

The workflow

1. Crawl the site and read the pages report rather than the issues report. Bloat is not an issue type; it is a pattern in URLs, titles, and word counts.
2. Sort by title. Repeated titles across many URLs are where templated duplication lives. Sort by word count next, because the low end is where thin templated output collects.
3. Take a sample of 10 suspect URLs and inspect them in Search Console.
4. Act on the verdicts, not the crawl. Unknown and duplicate URLs need a sitemap fix. Indexed-but-unwanted URLs need noindex or a 410.
5. Re-inspect the same sample in 30 days. Index changes are slow and a re-check is the only way to know the change landed.

Run this with the OpenSEO MCP

get_audit_pages gives you the crawl side and inspect_urls gives you Google's side, so an agent can join them in one pass. Both are free calls: they read stored audit state and Search Console rather than a paid data provider.

Using the OpenSEO MCP on project [name]:

1. From the most recent site audit, pull all crawled pages. Group
   them by exact title and show me every title used by more than
   two URLs, with the URL count and the word-count range.

2. Do the same for URL patterns: show me any path segment or query
   parameter that appears on more than ten URLs (things like
   /answer/, /page/, /tag/, ?filter=).

3. Pick the ten worst-looking URLs from those groups, weighted
   toward the largest group, and run inspect_urls on them.

4. Report each URL with its coverage state and its Google-selected
   canonical. Then sort them into: unknown to Google, canonicalised
   away, crawled but not indexed, and indexed.

5. Tell me the count in each bucket and what fraction of the
   suspect set is in the index. Recommend an action per
   bucket, and say plainly if the answer is that there is nothing
   to remove.

Point five is the instruction that keeps this workflow honest. Ask an agent to find index bloat and it will find index bloat. Ask it what fraction of the suspects are in the index and you get a number that can come back as zero.

Deletion is a scale decision

Both stories in this guide are true and they point in opposite directions, which is the useful part.

At 110 million pages with a measured 60% traffic loss, removing 5 million near-duplicates was the intervention that worked, and the agency that would not be aggressive enough had already failed at it. At 867 pages with no traffic collapse, the same instinct would have burned a weekend removing URLs Google had never fetched.

What separates them is not the size of the mess in the crawl report. It is whether the mess made it into the index and whether anything measurable got worse. Establish both before you delete a page, and the audit becomes evidence instead of an argument.

If part of your crawl came back blocked, hold this workflow until that is fixed, because a blocked page reports no title and zero words and will look exactly like thin content.

Index bloat FAQ

How do I know if my site has index bloat?

Compare the number of URLs you want indexed against the indexed count in Search Console's page indexing report. A large gap is the signal. Then inspect a sample of the surplus URLs, because a crawler can show you hundreds of near-duplicate pages that Google never fetched, which is not bloat.

Does index bloat hurt rankings?

It can, at scale, when a large share of a domain's pages are thin or duplicated and the domain gets assessed as a whole. On a site of a few hundred pages the more common effects are wasted crawling and internal link equity spread across URLs that were never going to rank. Neither is urgent on its own.

Should I noindex or delete duplicate pages?

Use noindex when the page has a purpose for users, such as a filtered listing or a paginated archive. Use a 410 or a 301 when the page has no purpose at all. Removing a URL from the sitemap alone does not deindex it; it only stops you asking for indexing.

How many pages should a website have indexed?

As many as there are distinct things worth ranking, which for most small business sites is dozens rather than thousands. The count matters less than the ratio: if most of your indexed URLs get no impressions in a year, the set is larger than the site can support.

Can a site audit tool find index bloat?

Not directly, because every bloated page returns 200 and passes its per-page checks. What a crawler gives you is the raw material, repeated titles, thin word counts, and recurring URL patterns. Pair it with Search Console URL inspection to find out which of those URLs Google holds. OpenSEO does both: crawls up to 50 pages on the free plan and 10,000 on the $10/month plan, and runs URL inspection against your connected property at no credit cost.