I Gave Claude Code a Brain. Then I Measured What It Costs to Ask for One.

At the end of part 2 I promised a clean competitor comparison: more than one memory system, the same admission gate, enough capacity to finish. This is that run. 1,453 agent sessions, four arms, five corpus conditions, every outcome graded by executing a checker against the code the agent actually wrote. No LLM judge anywhere. The corpus Each condition is a 4,911 document corpus, about 27,000 embedded chunks. Of those documents, 4,902 are adversarial distractors generated from the corpus's own vocabulary to be confusable, and the rest are the planted material for 36 tasks. That ratio is the point. On the small feed I used earlier, retrieval hit@10 was 1.000: everything was findable, so the benchmark could not tell "the product retrieved badly" from "the agent never searched". With the haystack, BM25 hit@1 drops from 0.485 to 0.182. The five conditions differ in what is planted, and four of the five are traps: | condition | what the corpus holds | share of cells | |---|---|---| present | the governing fact, cleanly | 37% | superseded | the fact, beside an outdated version of itself | 15% | adjacent | something related and inapplicable | 16% | absent | the fact deleted, distractors left behind | 17% | contradictory | two rival versions, no authority between them | 15% | Only 37 percent of cells contain a clean fact worth retrieving. In the other 63 percent the corpus is outdated, contradictory, inapplicable, or empty, and a memory layer's job there is to not make things worse. I built it that way deliberately. A benchmark where memory can only help is a demo. The control that changed the question Two commenters asked for a placebo. Fair: if you tell an agent "you have memory, consult it before acting" and it then does better, you have not separated the memory from the telling. So I built an arm carrying the complete memory instruction and no memory at all. Same protocol text, byte for byte, as the two real arms. Nothing behind it. | comparison | wins | losses | net | p | |---|---|---|---|---| | instruction only, vs bare | 26 | 43 | -17 | 0.053 | Telling an agent to consult memory, when there is no memory, makes it measurably worse. Seventeen cells of 358, borderline significance, in the direction nobody advertises. That reframes everything. "Does memory beat no memory" is not directly answerable, because every memory product requires that instruction. The answerable question is: does the product earn back the cost of asking for it? The comparison RE-call 0.11.0 is mine, so read accordingly. MemPalace 3.8.0 is the better known of the two. Both received the same instruction and the same gate. Every write tool was withheld from every arm. | comparison | wins | losses | net | p | |---|---|---|---|---| | RE-call vs instruction only | 41 | 21 | +20 | 0.015 | | MemPalace vs instruction only | 25 | 23 | +2 | 0.885 | | RE-call vs MemPalace | 48 | 30 | +18 | 0.054 | | RE-call vs bare | 47 | 44 | +3 | 0.834 | | MemPalace vs bare | 38 | 53 | -15 | 0.142 | McNemar exact on discordant pairs. The only comparison clearing p /VENDOR_REVIEW.md is for. Open an issue and I will help wire it up. And if you just want to check my arithmetic, the README has a section on verifying a published run with no credentials and no money. The preregistrations are in the repo, including the ones whose predictions I got wrong, and the raw records are published with each run. If you take one number from this, take that one: 63 percent of the cells were built to make memory a liability, and the memory layer still finished ahead of not having one. One thing I would most like broken: the 45 percent. On superseded , the largest single loss is that in 45 percent of sessions neither planted document is retrieved at all. That is worth roughly three times more than anything I have measured on the ranking side, it is a query-formulation problem rather than a retrieval-quality one, and it is already preregistered. If someone has a better idea than mine, I would like to lose that one. Top comments (0)