Google indexed 428 of my 1,432 generated pages and none of my 2,290 handwritten ones

I spent yesterday planning to delete the best part of my site. I run a calculators-and-tools site, solo, five months old. About 5,800 pages across four language sections. Google has 642 of them in its index. Not great. The rest sit in two buckets: 1,153 "crawled, currently not indexed" and 4,161 "discovered, currently not indexed." Like everyone in that situation, I had a theory. Somewhere in those 5,800 pages I had generated 1,432 timezone pages - every URL a pair of cities, /date-time/time-difference/london-and-buenos-aires , that sort of thing. One calculation, different arguments, spread across a URL space. It looks exactly like what Google's spam policy calls scaled content abuse, and I assumed it was dragging the whole host down. The plan was to collapse them into one hub page and 301 the rest away. Before doing it, I exported the list of what Google actually has. The export This is the part I had never bothered with, and it takes about four minutes. In Search Console: Indexing → Pages, then the link "View data about indexed pages." At the bottom of the examples table, change rows-per-page to 500. The examples table caps around 1,000 URLs, so if your indexed count is under that, you get the complete list, not a sample. Mine was 642, and I pulled 638 unique URLs out of it. Then classify them. I did it with a throwaway script that buckets by URL prefix: import re, collections def family(url): p = re.sub(r'https://example.com/[a-z]{2}', '', url) or '/' if re.match(r'^/date-time/(convert|time-difference|meeting-time|world-clock)/', p): return 'generated timezone pairs' if p.startswith('/recipes'): return 'handwritten recipes' if p.startswith('/cooking'): return 'cooking guides' if re.match(r'^/finance/[^/]+/.+', p): return 'generated finance scenarios' return 'tools and hubs' urls = [l.strip() for l in open('indexed.txt') if l.strip()] for name, n in collections.Counter(map(family, urls)).most_common(): print(f'{n:5d} {name}') Do the same for the "crawled, not indexed" and "discovered, not indexed" drilldowns and you get the whole picture in three exports. One trap if you automate the reading: the console reuses its DOM between drilldowns. My first pass at the "crawled, not indexed" list came back looking almost identical to the indexed list, and it took a set intersection to notice that 638 of those 660 rows were the indexed list, still sitting in the page. Reload the drilldown URL directly before reading it. What came back | Family | Total | Indexed | Rate | Crawled, rejected | Never crawled | |---|---|---|---|---|---| | Generated timezone pairs | 1,432 | 428 | 29.9% | 647 | 24 | | Generated finance scenarios | 180 | 74 | 41.1% | 65 | 4 | | Cooking guides | 530 | 61 | 11.5% | 36 | 95 | | Tools and hubs | 1,377 | 65 | 4.7% | 45 | 407 | | Market-specific pages | 593 | 8 | 1.3% | 0 | 138 | | Handwritten recipes | 2,290 | 0 | 0% | 0 | 195 | The two generated families I was about to delete are 502 of the 638 indexed pages. Seventy-nine percent of my index. They also have the two highest acceptance rates on the site. The 2,290 recipes - the ones with photographs, per-serving nutrition, ingredient scaling rules, written natively in two languages rather than machine-translated - have zero pages in the index. Not rejected. Zero of them appear in the "crawled, not indexed" bucket either. Google has never fetched a single one. (The counts in the last two columns are from samples - those drilldowns cap out around a thousand example URLs, and my buckets are larger than that. The indexed column is complete.) Why this is not the story I expected The obvious reading is "Google likes generated pages better," which is nonsense. Look at the last two columns instead. For the timezone family, Google is finished. It crawled essentially all 1,432, kept 428, threw out 647, and has almost nothing left in the queue. That is a completed judgement, and 30% is the verdict. For the recipes, Google hasn't started. They are all sitting in "discovered, currently not indexed," which does not mean rejected. It means Google knows the URLs exist and has not spent a request on them. Those are two completely different failure modes and the summary number blends them into one scary total. "Crawled, not indexed" is a quality verdict. "Discovered, not indexed" is a scheduling decision - Google deciding your host isn't worth more requests right now. My Links report says external links: 0. That's the whole story: crawl allowance got spent on whatever was linked from the hubs first, and the rest of the site never got its turn. So the site does not have a content-quality problem in the family I suspected. It has a "nobody links to this domain" problem, which no amount of deleting pages will fix. What I would have lost Had I shipped the collapse-and-redirect plan, here is the sequence. Day one, the old URLs are still in the index. Someone searches for a city pair, finds the old URL, clicks, gets a 301 to the hub with the pair pre-filled. Fine - the reader gets their answer. Week four, Google re-crawls, sees the redirects, drops the old URLs and shows the hub. Now one generic hub page has to rank for every city-pair query on its own. A hub with two dropdowns competes badly against a page whose title is literally the query. Net effect: hand back 67% of the index in exchange for rankings the hub would have to earn from scratch. To fix a quality problem that the data says is not there. The actual takeaway If you are about to prune pages because they feel thin, export the indexed list first. Four minutes. The report you need is one click below the chart most people stop at, and the answer may be the opposite of your theory. Mine was. And split those two "not indexed" reasons in your head permanently. If your pages are crawled and rejected, work on the pages. If they are discovered and never crawled, the pages are irrelevant - go get links, because you are asking Google for more of its budget than your domain currently earns. I am doing the second one now. If it's useful to anybody, the recipe data those uncrawled pages are built on is published as an open dataset - 501 dishes from 127 countries, JSON and CSV, CC BY-SA 4.0, with per-serving nutrition and sourced cooking times: theunitools.com/en/data. Google hasn't read it either, but you might. Top comments (0)