They knew it wasn't the model. They patched it anyway.

They Knew It Wasn't the Model. They Patched It Anyway.

Two articles ago, I described a habit: when a system has a language model inside it and the output wobbles, the explanation drifts to the model. Then I measured it, and the interesting result wasn't the blame - it was the behaviour.

Without access to the code, nineteen of twenty agents set about damping the output instead of looking for the cause. This piece was supposed to answer why. I had two hypotheses and they made different predictions, which is the good kind of problem to have. Instead, I ran the control first, and the control made both of them pointless.

The Control

The design is almost embarrassingly simple. Take the same classifier, the same planted fault, the same corpus, the same five passes, the same trace format. Change one thing: the head that does the classifying. In one arm, it's a language model. In the other, it's a random forest - trained by distilling the model's own labels, frozen into a pickle, and given the identical interface so that nothing else in the system differs by a single byte.

The whole control. The fault sits in retrieval, upstream of the head, so it is the same fault in both arms - and the head, whichever it is, classifies correctly whatever it is handed. Both briefs carry the same measured certification, and it is the piece that makes the comparison fair: "re-running the classification over the same context reproduced the output in 260 of 260 cases" - the real number, identical in both arms.

The Results

Without the code, forty agents saw the table without the code, twenty per arm. And then another forty, to find out whether what came back was the effect or the sample.

| What Happened | Language Model | Random Forest | p |
| --- | --- | --- | --- |
| Patches the symptom | 19/20 | 19/20 | 0.76 |
| Accuses it of randomness of its own | 14/20 | 4/20 | 0.0018 |
| Proposes taking the randomness out | 17/20 | 11/20 | 0.041 |
| Blames the head | 13/20 | 6/20 | 0.028 |
| Uses the determinism argument | 9/20 | 16/20 | 0.024 |
| Places the cause upstream | 16/20 | 17/20 | 0.50 |
| Finds the real cause | 2/20 | 8/20 | 0.032 |
| Asks for the data it lacks | 0/20 | 0/20 |

The Dissociation

Of the twenty-five responses that exonerated the head - across both arms, on the same argument and the same certification - twenty-three proposed a patch anyway.

Measured Twice

Forty responses are forty responses. Before these forty, there are another forty, over the same two packages without a byte of change, and they're what tells effect from sample.

The Nature of the Suspicion

What changes is that they get less far: two of twenty against eight find the cause, and the same two against eight name the real mechanism. A small, consistent difference - not the one I expected.

The Question that Lost Half its Object

I came into this piece with two hypotheses about why the reflex exists. One said it was a fossil of the training corpus - a habit from an era when temperature really was the main dial and treating output variance as a property of the model really was correct. The other said it was the model's personality, some being more inclined than others to look outward before looking at their own work.

What Doesn't Hold

Looking upstream doesn't replicate, so I claim nothing about where the search stops. It's the row I'd most have liked to keep. The difficulty band broke in the second sample. The pre-committed criterion allowed up to four of twenty of difference in finding the cause; it came out exactly four the first time and six the second.

The Honest Reading

The honest reading splits the thing I'd been calling one behaviour into two, and only one of them is generic. Patching is generic. Nineteen of twenty on each side, in both samples, head exonerated or not. Investigating barely moves. They look upstream equally.

The Nature of the Suspicion

What changes is that they get less far: two of twenty against eight find the cause, and the same two against eight name the real mechanism. A small, consistent difference - not the one I expected.

The Process Note

The first version of this control didn't certify the two arms alike: 260/260 for the forest and 240/260 for the model. Since the argument for ruling out the head is one of the things being measured, making it more available on one side contaminated exactly what mattered.

The Most Robust Thing

Across five scenarios, two kinds of head, passive permission and explicit permission, not one of two hundred and eighty responses has asked for the information it was missing before concluding. That number has survived every manipulation I've thrown at it, including the one designed to break it, the one that took the language model out of the loop, and the one that repeated the whole measurement from scratch.