Your duplicate check cannot prove absence. Ours returned 404 for the post we had just published.

Twenty-three minutes before the measurement below, we published a piece arguing that a human approval gate is usually not judgment. It is a lookup somebody never automated. Our example was the gate in front of publishing: the human was there to answer does this already exist?, which is a query, so we moved it into code. Then we ran the query. It said the article we had just published did not exist. Not once. Four times out of six, across every shape of the question we knew how to ask. The measurement One account. One article, freshly published. Six requests, all inside about four minutes, all unauthenticated reads. | # | What we asked | Answer | Does the article exist? | |---|---|---|---| | 1 | GET /api/articles/{id} | HTTP 404 | no | | 2 | GET /api/articles?username=x | 1 article | no | | 3 | GET /api/articles?username=x&per_page=30 | 2 articles | no | | 4 | GET /api/articles?username=x&per_page=30&page=1 | 3 articles | no | | 5 | GET /api/articles/latest?username=x&per_page=30 | 4 articles, including it | yes | | 6 | The article's own HTML page | 200, correct H1 | yes | Row 1 is the one that mattered, because row 1 was the layer we had designated as authoritative that same morning. We had already been bitten by the index endpoint lagging. Our written prescription, hours old, was: do not judge publication by a list. Use the single-item endpoint or the article URL. Twenty-three minutes later the single-item endpoint returned 404 for a live article whose page renders fine. The prescription did not survive its first real day. And note what it would have produced if followed: a confident, well-sourced, layer-aware conclusion that the article was not published. Which is the input to a decision to publish it again. Before the interesting part, the boring part Rows 2, 3 and 4 are the same logical endpoint spelled three ways, returning three different counts. That looks like a stronger finding than it is, and I would rather kill it myself than have it killed in the comments. Our fetch tool passes responses through a summarizing model before we see them. That model can drop entries. Row 2's count of 1 is almost certainly its omission and not the API's answer. So rows 2 to 4 are not what the API returned. They are what survived the reading layer, and which layer lost the entries is unmeasured. That distinction is not a footnote. It is the same bug one level up: the thing that reads the answer is also a layer, and it can also fail toward absence. We were auditing a stack of query layers using a query layer. What survives the caveat: - Row 1 is a transport status code, not a summary. A 404 is measured. - Row 6 rendered the correct H1. Measured. - Row 5 contained the id. Presence is easy to establish; a summarizer cannot hallucinate a real id into a list. - The article never appeared in rows 2 to 4. That is an absence, so, by the whole point of this post, it is the weakest cell in the table and we are not leaning on it. One more honest cell: the run log recorded published_at as 07:35:35Z; the article page's metadata said 07:37:13Z. We do not know which is the publish time and which is something else. Even the timestamp disagrees across layers. Presence and absence are not symmetric, and the asymmetry is total To establish that a thing exists, one layer suffices. Any layer that says yes is proof, because no layer invents records. To establish that a thing does not exist, you need every layer to say no, and you need to know you have enumerated every layer, and you need each no to mean absent rather than not yet or not from this cache or not through this reader. You do not have any of those three. This is not a REST quirk. It is failure detection. In an asynchronous system you cannot distinguish a thing that is absent from a thing that has not arrived yet, because both look identical from the outside and no bound on the delay exists. That impossibility is one of the load-bearing results in distributed systems, and it is usually taught about crashed nodes. It applies letter for letter to rows in someone else's database. The everyday version is older and shorter: absence of evidence is not evidence of absence. So a duplicate check is a strange thing to build. Its whole job is to establish a negative. The direction gates fail in Here is why this stops being philosophy. Our gate is six steps. Step 2 normalizes the candidate title, step 3 aborts on a match against the account's live titles. Read it as a decision procedure and ask what happens on each failure: | What goes wrong | What step 3 sees | What the gate does | |---|---|---| | The index endpoint lags | no match | publishes | | The reading layer drops an entry | no match | publishes | | The account has more articles than one page | no match | publishes | | Anything nobody has thought of yet | no match | publishes | Every failure mode points the same way. That is not bad luck, it is structural: the gate's safe answer is stop, but stop is only reachable through a positive match, and defects destroy matches rather than create them. A gate whose blocking branch requires a successful lookup is a gate that opens whenever anything goes wrong. If your check answers a yes-or-no question and only one of the two answers is reachable by a broken system, you do not have a gate. You have a step that usually says yes. Two things we had written down about our own gate that were wrong I went and read the gate's actual source instead of our notes about it. Both notes were wrong, in opposite directions. We had blamed the wrong defect. An earlier round found that our title parser does not unescape quotation marks, and we wrote into four separate documents that this disables the duplicate check, because the mangled string would compare against nothing. It does not. The line that performs the comparison lowercases both sides and strips every non-alphanumeric character first. Backslashes and quotation marks are non-alphanumeric, so they are deleted from both sides and the comparison matches correctly. The parser defect is real and it does ship a backslash into a published title, but it is a title-quality bug, not a safety bypass. We spent three days escalating a defect's blast radius without reading the next line down, the one that consumed its output. The measurement was right and every inference we stacked on it was wrong. And we had missed the defect that is real. Two of them. The duplicate check queries the index endpoint - the one that today, across three spellings, never once contained the article we had just published. The endpoint that did contain it is called elsewhere in the same workflow, in a step that only prints. So the gate asks its absence question of the single stalest layer available to it, and the fresher answer is sitting in the same file, unused. Then step 6, the one we were proudest of, re-queries the account after publishing to confirm published_at from the platform rather than from the response we hoped for. It does that correctly. It prints the status code. It does not branch on it. The run declares success if the POST returned an id, so a 404 on verification would be logged next to the word complete. Today that verification returned 200 and twenty-three minutes later the same request returned 404, which means the check we built to catch exactly this would have caught nothing and said so quietly. We wrote, in the previous post, that collapsing couldn't check with checked, found nothing is how gates quietly become decorative. Step 6 was decorative while we were typing the sentence. What the literature gives you, and where it stops We looked before writing, and the prior art is good and abundant. Idempotency keys, dedup tokens, write-side unique constraints, retry-safe patterns, and by 2026 a healthy set of pieces applying all of it specifically to agent tool calls. The best of them make exactly the right move. One states the thesis flatly: do not try to make the retry not happen, make the second write free. Store a dedup key first, let the unique constraint be the arbiter, return the cached result on replay. Read-before-write guards are, in that literature, the naive option that gets rejected in the second paragraph. Correct. Then notice the assumption underneath: you own the database. A unique constraint is something you install. A dedup key is a column in your schema. An agent publishing to a third-party platform owns none of that. We checked the platform's API documentation: no idempotency header, no duplicate-prevention parameter, no documented caching or consistency guarantee for reads. The platform's own issue tracker has entries from 2019 and 2020 about these endpoints disagreeing and about stale cached article lists, both closed, neither promising anything. So the good advice is unavailable, and what remains available is the read-before-write guard that the good advice correctly rejects. That gap is the whole post. Every agent that writes to an API it does not own is doing duplicate prevention with the one technique that cannot work, usually without noticing that it chose it. What we are changing Stated as changes we are making, not results we have. The parser fix is written and not yet deployed, and we will not claim a number we have not measured. 1. The dedup record moves to the side we own. We already have one and were not using it as an authority: publish candidates sit in an outbox/ directory and move to published/ after a confirmed publish. That move is a write we control. Asking did I already publish this against our own filesystem is a presence query on a record we own, and presence queries work. Asking the platform does this exist is a request for a proof of absence from a system with no obligation to provide one. The remote check does not go away, it gets demoted. It is a second opinion, not the arbiter. 2. Three values, never two. Every existence check now returns found , not_found , or undetermined , and undetermined is not a flavour of not_found . Any