My Release Gate Passed. The Model It Shipped Answered 'Neutral' To Everything.

Setup

I trained a model that scored 4 percentage points better than the one I ship. It's gone. I deleted it from the roadmap yesterday. That's not the interesting part. The interesting part is how I found out it was broken - because it passed every check I had, and then shipped a file that answered neutral to every question you could ask it.

I maintain semantix, a local NLI judge for compliance validation. The shipped model, nli-popia-v2, is an 82M-param cross-encoder fine-tuned on clauses of South Africa's POPIA. It runs as INT8 ONNX on CPU - ~15-50 ms depending on text length, no API key, no data leaving the machine. Determinism is the whole point - same input, same score, every run, because "rerun that validation" is a question a regulator actually asks.

Its model card has said this for months: 82M-param base. A larger base would likely improve in-domain F1. v2 retained the small base for ONNX deployability (~79 MB quantized).

The bigger model

So: obvious next move. Try a bigger base. deberta-v3-base - 184M params, 2.25× larger. Train it, measure it, ship it. The bigger model won. It trained clean on an L4 for about fifteen cents. Against the pinned holdouts:

The hypothesis in the model card was right. Bigger base, better F1, on both holdouts, same eval script, same pinned data. The release gate passed. I published the repo.

Everything above is true and every number is real. The model I shipped was still broken.

The file that shipped

Someone loaded the actual artifact - onnx/model_quint8_avx2.onnx, the file the library downloads - and ran it against the canonical failing example from my own demo:

"Our signup flow presents a pre-ticked marketing opt-in checkbox that customers must untick if they don't want newsletters."
vs.
"The responsible party is obtaining explicit, informed, voluntary consent…"

A pre-ticked box is the textbook contradiction of freely-given consent. v1 knows this:

v3 said neutral, hard.

Debugging label ordering

My first instinct was the worst possible one: label ordering. I'd shipped exactly that bug once before - reading probs[2] (neutral) where probs[1] (entailment) belonged - and it produced a negative correlation in a benchmark I nearly published as a finding. I wrote that one up too.

So we tested it properly: score the artifact on 24 stratified pairs from its own bundled eval set, 8 per class, under all six possible index→label mappings.

0:contradiction 1:entailment 2:neutral  macroF1=0.1667   <-- the declared mapping
0:contradiction 1:neutral 2:entailment  macroF1=0.1667
0:entailment 1:contradiction 2:neutral  macroF1=0.1667
0:entailment 1:neutral 2:contradiction  macroF1=0.1667
0:neutral 1:contradiction 2:entailment  macroF1=0.1667
0:neutral 1:entailment 2:contradiction  macroF1=0.1667

pred index counts: {0: 0, 1: 0, 2: 24}

Six identical numbers. That flatness is the diagnosis. 0.1667 is exactly what macro-F1 degenerates to when a model emits one class for everything on a 3-way balanced set: one class scores F1 0.5, the other two score 0, and 0.5/3 = 0.1667. Permuting the labels of a constant just renames the class it always guesses. The table is flat because there was no signal to recover. It wasn't label ordering. It was a dead model wearing a live model's config.

Why the gate didn't catch it

Here's the uncomfortable bit. My release gate was real. It enforced macro-F1 deltas and per-clause non-regression, in CI, on every push. It scored 0.7850. That number was never wrong. It scored the PyTorch weights.

The pipeline is: train → gate → export to ONNX → quantize to INT8 → upload. The gate sat before the two steps that produce the file anyone actually downloads. Export and quantization are transformations, and I had zero checks after them.

So the gate could pass - honestly, correctly - on a model whose shipped artifact was a constant. I had built a validation you couldn't reproduce from the thing being validated. Which, given that my entire product exists because "the LLM said it was fine" isn't a defensible answer, is a special kind of embarrassing.

The actual cause

deberta-v3's disentangled attention is sensitive to per-tensor INT8 dynamic quantization. The activation ranges across its attention projections don't survive a single scale factor per tensor - the logits collapse and the model saturates to one class. v1 and v2 are roberta. Roberta quantizes cleanly. I'd never hit this because I'd never shipped a DeBERTa. The moment I changed the base, a step I had never needed to think about became the step that broke everything.

The fix and the real answer

The fix for the collapse is one argument: AutoQuantizationConfig.avx2(is_static=False, per_channel=True). Per-channel scales instead of per-tensor. That stopped the constant predictor dead.

With per-channel quantization, v3 works. It predicts real classes. And here's what it scores:

Quantized v3 (0.576) is far worse than quantized v2 (0.7465), the model it was supposed to replace. The new artifact gate - the one that scores the shipped file - blocked the upload automatically. First real outing, and it earned itself.

So the honest summary of the whole experiment: v3's advantage exists only at fp32, at 704 MB. Under the INT8 quantization my product requires, v3 is worse than what I already ship. deberta-v3-base is a bad fit for a quantized-CPU-judge line. Not a bug. A property.

That old line in the v2 model card - "v2 retained the small base for ONNX deployability" - was a hunch when I wrote it. Now it's a measured result. The bigger base really is better, and it's unusable under the constraint that actually binds. I'm glad I know that for a reason instead of a feeling.

What actually changed

Two things, both now on master and shipped in 0.2.3:

1. The gate scores the artifact. It loads the quantized ONNX through onnxruntime, scores it on the same holdout, and asserts macro-F1 within tolerance of the PyTorch gate.
2. It counts distinct predictions. This is the cheap one:

def onnx_macro_f1(onnx_path, tokenizer, eval_rows, batch_size=16) -> tuple[float, int]:
    """Macro F1 for a quantized ONNX artifact + number of distinct predictions."""
    ...
    len(set(preds)) > 1

One line. It catches a constant predictor in seconds. Everything above - the permutation sweep, the config archaeology, the tokenizer forensics - would have been unnecessary if that assertion had existed from the start.

The thing worth taking

Your release gate probably validates a model. Ask whether it validates the file you upload. If there's any transformation between them - export, quantization, conversion, pruning, a format change - then your gate and your artifact are two different objects, and the gate is measuring the one nobody downloads. Mine passed with an honest 0.785 while shipping something that couldn't tell consent from a contradiction. A model can pass its own release gate and ship dead. Mine did.

Postscript

In the interest of not overclaiming: v3 also never beat v1 on v1's own 7-clause holdout - 0.785 against v1's 0.813. v1 is a 7-clause specialist; v2 and v3 are 10-clause generalists, and broader coverage costs in-domain accuracy. The bigger base bought back about 4 of the ~7 points v2 gave up. It didn't repay the debt. That's in the v2 model card too, and it should travel with any sentence that starts "the newer model is better."

semantix is MIT, the models are Apache-2.0, and every holdout mentioned here is pinned and downloadable, so you can disagree with me from the data: github.com/labrat-akhona/semantix-ai