DEV Community 11h ago

You Deleted It. The Crawler Still Sees It.

I removed two listings from a directory site I run. Not for a small reason - they failed the one promise the directory exists to make. The kind of removal you do immediately and double-check.

I did double-check. Flipped the flag in the database, loaded the site, watched the queries filter them out. Gone from the city pages, gone from the main directory. Removal confirmed.

A day later, both were still on the site. Not on some cached page or stale CDN edge. In the current, freshly deployed HTML - sitting in the exact lists I had watched them disappear from.

I had verified the deletion in one document. The site is two.

A rendered site is two documents

Here's the architecture, because the bug lives in it. The site is a single-page app: the browser downloads a JavaScript bundle, the bundle fetches data, the page assembles itself client-side. Crawlers are historically bad at waiting around for that, so - like a lot of SPA operators - I run a prerender step at build time. A script fetches the same data, bakes real content into each route's static HTML, and that is what robots, link previews, and anyone with broken JavaScript actually receives. The bundle takes over afterward and re-renders everything live.

It works. It's also a decision with a consequence I under-appreciated for months: every fact on the site now exists twice.

Once in the live render, computed in the browser from a fresh query.
And once in the baked copy, computed at build time by a different script, from a different query, through a different code path.

Two documents. Two authors. One URL. And the two authors don't consult each other.

The browser filters listings with a WHERE allowed = true. The build script - written earlier, for a different purpose - never selected that column. So when I "deleted" the two properties, the browser's query dutifully dropped them, and the build script kept baking both into the static HTML on every deploy, complete with links and ratings. From its perspective, the deletion had never happened.

The deletion was real in one document and fiction in the other. Nothing errored, because nothing was wrong: both documents were internally consistent, both rendered cleanly, both passed every check that looks at documents one at a time.

Three divergences, one day

Once I knew to look for the split, I found it three times in a single day - same site, escalating stakes.

The title fix that wasn't. A page was earning search impressions for a phrase its title didn't contain, so I added the phrase. Checked the page in the browser: there it was, in the tab, in the markup. But client-side titles are set by the bundle after load - the crawler reads the baked <title>, and the baked title still didn't have the phrase. The fix was fully live for every human and invisible to the one reader that determines whether the page ranks. It had been "fixed" in the wrong document.

The labels that didn't propagate. I'd spent that week making the site more honest - replacing a blanket "verified" badge with per-listing labels that say what's actually confirmed versus merely claimed. The app rendered the new labels beautifully. The baked copy went right on telling crawlers every listing was verified, in a header the build script had been stamping onto nine different pages. The honesty overhaul shipped to the browser and not to the record.

The deletion. The first two failures overstate; this one resurrects. A removed listing isn't a stale adjective. It's an entity I had decided, for cause, should not be presented - still being presented, by a document I'd forgotten was the one that counts.

And the failure runs in both directions. While fixing the build script I added a column to its fetch that didn't exist in that table. The data API rejected the query - and the script, hitting an error, quietly returned an empty list, at which point every detail page fell back to a generic template. No build failure. No warning. Just a deploy where the baked document silently regressed while the live one stayed perfect.

Two documents means twice the ways to be wrong, and each one's failures are invisible from inside the other.

Why nothing catches this

Every tool I had was pointed at one document or the other. Never at the seam.

The type checker checks the app.
The tests exercise the app.
The browser - where all manual verification happens, because that's where you look at your site - runs the app.

Meanwhile the build logs confirm the prerender ran, which is not the same as confirming what it wrote. The bug wasn't in either document. It was in the fact that there were two.

This is the oldest failure shape in data systems - the same fact stored in two places will drift, given time - wearing a disguise that makes it easy to miss: the two copies here aren't two database rows or two config files. They're two renderings of the same URL, which is precisely why it doesn't feel like duplication. It feels like one page. You'd never think to diff a page against itself.

"Done" names a document

The fix for the specific bugs was mundane: make the build fetch select the flag, filter on it, propagate the labels, correct the title - in the baked layer, where each change should have landed first.

The durable fix was changing what I accept as evidence that a change shipped. Looking at the site in a browser verifies the live document. It verifies nothing about the baked one. So the definition of done, for anything that matters to a crawler - a title, a claim, a listing's existence - is now mechanical: fetch the served HTML the way a robot would, with no JavaScript executed, and confirm the change is in the bytes.

curl, grep, done. Ten seconds. It would have caught all four incidents before deploy, and it's the only check that examines the document I kept forgetting existed.

If your site renders twice, you don't have a page. You have a page and its understudy, and the understudy is the one performing for the audience that decides whether anyone finds you. Verify the performer that's actually on stage.

You Deleted It. The Crawler Still Sees It.