TopMovieList is a movie, actor, and director database we build and run. It generates pages from a Postgres database, caches them in Redis, and serves them through Next.js on the edge. By the raw numbers it looked healthy: tens of thousands of pages, structured data on everything, fast responses. Then I opened the Page Indexing report in Google Search Console and found the number that started this whole project.
3 pages indexed. 9,921 not indexed. The reason for almost all of them was a single line: "Discovered, currently not indexed." No manual action. No crawl error. No stray noindex tag or broken canonical. Google had found close to ten thousand of our URLs, looked at the site, and decided it was not worth the crawl budget to index them.
What follows is the diagnosis and the fix. The results are a separate post, because index coverage moves in weeks and it would be dishonest to sell this as a turnaround before the numbers move. Call it part one: the reasoning, and the changes that came out of it.
What "Discovered, currently not indexed" actually means
That status is Google saying it knows a URL exists but has chosen not to fetch and index it. It is not a bug you fix with a redeploy. It is a judgment about quality and trust. On a normal site you might see it on a handful of thin pages. Seeing it on 9,921 URLs with 3 survivors is a different message. Google was rejecting the site wholesale.
The uncomfortable part is that this is exactly the response Google's spam and helpful-content systems are built to give programmatically generated pages at scale. TopMovieList leans heavily on AI-written summaries. Bing indexed and ranked the same pages without complaint. Google would not touch them. That gap between the two engines was the real tell.
Reading the actual shape of the site
Headline page counts hide the problem, so I went to the database and counted what actually feeds the sitemap. The generator pulls every row where noindex = false, which means the sitemap size is really just a query result:
- Movies: 9,666 rows, 5,494 in the sitemap.
- Actors: 14,979 rows, 2,640 in the sitemap.
- Directors: 4,148 rows, 1,550 in the sitemap.
So roughly 9,700 of the 9,900 sitemap URLs were entity detail pages. Every one is the same shape: a single AI-written blurb about one movie, actor, or director, plus a generated image. The actor and director pages were the thinnest. A director page was one paragraph and a list of films, with nothing on it that Google could not already get from Wikipedia or write itself.
An earlier pass had already tried to prune this. More than half the actors were flipped to noindex, and Google still indexed 3. That told me the page count was a red herring. The template was the problem. Cutting from 9,900 near-identical blurbs to 2,000 near-identical blurbs does not change what they are.
Shrinking the indexable surface to something defensible
The plan was to stop asking Google to index anything that reads as templated filler, and keep only the pages a human editor would actually publish. It helped to notice that indexability on the site runs through two separate gates:
- A
noindexboolean column in Postgres, set by a backfill script. A page keepsnoindex = falseif it has real Bing search demand or clears a notability bar, which for movies is an IMDb rating at or above 7. - On top of that, a threshold on our TML Score, a 0 to 100 rating we compute for every movie as a blend of its IMDb rating and its Metascore. Among the indexable movies, only those at 85 or higher stay in the sitemap and keep indexing on. That took the movie set from 5,494 down to 409.
The threshold lives in one JSON file that both the sitemap generator (plain Node) and the movie page's metadata (TypeScript) read. That was deliberate. The fastest way to reintroduce this exact bug is to let the sitemap and the page disagree about which URLs are indexable, so they now read the same number from the same place.
The mechanics were unglamorous, which is the point:
- The sitemap generator got a
gte('tml_score_value', 85)filter on the movie query, and the actor and director enumeration loops were deleted outright. - Actor and director detail pages now return
robots: { index: false, follow: true }from generateMetadata. They stay live for users and still pass link equity through. They just leave the index. - The A to Z browse pages and the genre-by-letter permutation pages (things like
/genres/comedy/K) got the same treatment. They are navigation, not content.
The sitemap went from about 9,900 URLs to 568. Roughly 4,200 single-paragraph people-pages left Google's view in one commit.
Here is the honest cost of that. A global noindex does not care which engine is asking, so those pages left Bing too, and Bing had been ranking them without complaint. I was pulling pages that worked on one engine to fix a problem on another. I did it anyway. The Bing traffic to any single actor or director page was tiny, a few sessions scattered across thousands of URLs, and the pages Bing actually sent real readers to were the genre analyses and quizzes and best-of lists, which are editorial and stayed indexable. The alternative, serving a noindex only to Googlebot, is fragile and reads as cloaking, which invites a worse problem than it fixes. So I took the cleaner site-wide signal and ate a small Bing cost.
The same logic reached the ads. The site ran Google AdSense auto ads that had earned about five dollars in fifteen months and were the likeliest cause of a poor mobile layout-shift score. Serving ads that shove the layout around, on thin pages, while trying to prove those same pages are worth indexing, is the identical self-defeating move in a different spot, so I turned them off.
The part where a substring match bites you
With the filler gone, the next move was to grow the pages that are not filler. The site already had a template for "Every [Director] Movie, Ranked," and only 18 directors used it out of more than 4,000. So I queried for directors with a deep enough scored filmography and added 33 of them.
Before shipping, I checked the film lists those pages would actually render, because the page fetches a director's movies with a case-insensitive substring match:
ilike('director', '%' + name + '%')
Most names are distinctive enough that this is fine. Ron Howard is not. "Ron Howard" is a substring of "Byron Howard," so Ron Howard's page was about to list Encanto, Zootopia, and Bolt, three films by a completely different director. A couple of other names pulled in genuine co-directed credits, like Enemy of the State and Supernova, which happened to be right by accident.
The fix keeps the wide query but filters the results to rows whose director field lists the exact name as a comma-separated token. That drops Byron Howard's films from Ron Howard's page and keeps the real co-directing credits, because those genuinely contain the name as its own token. It reuses the exact-token check the genre pages already had, so there was no new machinery to introduce.
Why 85, and whether it even matters
An 85 is not different in kind from an 84. The cutoff is arbitrary at the margin, and I would rather say that plainly than pretend a movie scoring 85 is suddenly editorial. It is a volume lever. The bet underneath it is about ratio, not any single page: when 98 percent of your sitemap is one thin template, the site reads as a content farm and Google declines the whole thing. Cut the volume of that template and the pages that are actually different stop drowning in it.
The fair objection is that the pages I kept are also templated and also AI-assisted. The ranked-filmography pages come off the same kind of pipeline as the entity pages I cut. So is the problem the AI writing or the amount of it? My working answer is that the line is not templated versus handwritten, and not AI versus human. It is whether a page does something past restating one database row. An actor page wraps a single row: a name, a paragraph, a list. A ranked filmography or a "best underrated sci-fi" list pulls from many rows and makes decisions about order and inclusion, which is a thing a person might search for and a thing a bare data dump is not. That is a hypothesis, and the next month of coverage data settles it: if Google still refuses the curated pages, the problem was the AI content itself or the domain's authority, not the volume.
That authority question is the one most writeups put last and skip. A low-authority domain that publishes thousands of pages gets "Discovered, currently not indexed" by default, because Google will not spend crawl budget on a site it does not trust yet. Fewer, more differentiated pages help. Real links from other sites help more, and that work does not fit in a pull request.
So part one ends without a scoreboard, which is the honest place to end it. The sitemap is smaller and the surviving pages are a different kind of thing than the long tail Google rejected. Part two is the coverage report.