The Rubric Line That Sent Me Down a Three-Layer Debugging Rabbit Hole

Two days before my hackathon deadline, I re-read the judging criteria for the track I was submitting to and found a sentence I'd apparently skipped the first time through: "...a measurable efficiency gain over single-agent baselines."

My project, RedCouncil, is an adversarial multi-agent system - five specialized AI agents (Growth, Risk, Legal, TechDebt, Customer) debate a business decision, cross-examine each other, and a Synthesizer produces a severity-scored report. 68 tests passing, auth, storage, deployment, all done. I had never once measured whether it actually beat a single model call. My first instinct was to bail. My second, better instinct was to ask: how fast can I actually get real evidence for this, instead of assuming it's either fine or hopeless?

Building the eval, fast

The fastest honest version of this isn't a new subsystem - it's reusing everything you already have. Same scenarios, same model, one new prompt that asks a single Qwen call to do what all five agents do combined, and a purely deterministic scoring function - no LLM judge, just counting and set math.

Two numbers mattered most:

• Cross-agent conflicts surfaced. A single model call has no mechanism to represent disagreement between specialist viewpoints. This number is 0 for the baseline by construction, which makes it a free, structurally guaranteed data point.
• High-severity findings caught , compared head-to-head on the same 1-10 scale, same model, so the comparison isolates the architecture, not model size.

I ran it. Numbers came back. They looked good. I started building the slide deck. Then I actually read the results file instead of just the summary, and found the first problem.

Bug 1: the metric that was empty by design

One of the fields I'd built, domains_missed_by_baseline, was empty in every single scenario - which I initially read as "RedCouncil never misses anything the baseline catches." It took a second look to realize the field was always going to be empty, because of how I'd defined it: it only checked domains present in RedCouncil's output but absent from the baseline's. Since the baseline prompt forced one finding per domain every time, that set difference was structurally guaranteed to be empty regardless of how RedCouncil actually performed. Not a finding - an artifact of how I'd defined the metric.

Worse, when I checked the reverse direction, the one that actually mattered, I found RedCouncil was missing an entire domain - Growth - in 3 of 5 scenarios. My own multi-agent debate system was covering less ground than the single dumb baseline it was supposed to beat.

Bug 2: the retry that gave up too early

Root-causing this with real trace data (not speculation) showed a two-part failure: the coverage checker only flagged a domain as "missing" if its findings scored above a severity threshold - so a legitimately lower-severity Growth finding could slip through undetected. And when a retry did get triggered, the acceptance logic only kept the retry if it strictly improved on the previous attempt. If the model's retry attempt failed to recover the missing domain, the system gave up and kept the broken result - because "no better than before" and "still broken" looked identical to the acceptance check.

Fix: lower the threshold so every domain gets checked regardless of severity, and give the retry loop several bounded attempts instead of one, tracking convergence properly instead of a single strict comparison. I re-ran the comparison. Growth showed up in all 5 scenarios. I updated the deck. I thought I was done.

Bug 3: zero, again

Then I actually opened the live product UI to record demo footage, and the "conflicts surfaced" counter read 0 - on a scenario that, reading the transcript, clearly had agents contradicting each other's core claims in plain English.

Round one of debugging this found a genuine plumbing bug: one part of the code wrote a result to a field called conflicts, another part read from a field called conflict_count. Nobody had ever written to that second key. Classic. Fixed it, redeployed, tested again. Still zero.

Round two went deeper, and this is the part I actually learned something from: the conflict detector was gating candidate disagreements through a lexical-overlap check before it would even evaluate whether they opposed each other. My Growth agent argues in terms of savings, cost, ROI. My Customer agent argues in terms of churn, friction, retention. Zero shared vocabulary - so the gate filtered the pair out before the actual stance-comparison logic ever ran, even though the underlying disagreement was completely real.

This is the kind of bug that's obvious once you see it and invisible until you do: specialists with genuinely different mandates use genuinely different words by design. A similarity gate built for a system where everyone talks about the same thing is exactly the wrong tool for a system whose entire premise is that the agents don't think alike.

The fix: stop inferring disagreement from vocabulary overlap. Trust the signal the agents themselves already produce when they explicitly rebut each other during cross-examination.

Bug 4: the one hiding inside the fix

Trusting each agent's own rebuttal signal introduced a new failure mode almost immediately: rebuttals are often mutual. If Growth rebuts Risk and Risk rebuts Growth in the same exchange, and each rebuttal becomes its own conflict entry, you double-count every disagreement that goes both ways. Caught this one before it shipped, with a simple fix: dedupe on the unordered agent pair before counting, not the directional edge.

What actually changed by the end

Conflict detection went from a broken 0, to a plumbing fix that still read 0, to a real but partial fix, to a detector that could finally see disagreements that don't share vocabulary - at which point the true conflict count roughly doubled from what the vocabulary-limited version had found. Every one of those jumps looked like "the fix worked" right up until the next layer surfaced.

What I'd actually tell someone building an eval harness

• A metric that's always the same value isn't a metric - check what it's structurally capable of showing before you trust it. Mine was empty-by-definition for an entire category of finding.
• If your system's whole premise is that components think differently, don't build detection logic that assumes they'll describe things similarly. That's not an edge case, it's the design working as intended running into a detector that assumes the opposite.
• Fixing your eval can be as much engineering work as fixing your product - arguably more, because a broken product fails loudly and a broken eval fails silently, as a confident, wrong number.
• Non-determinism compounds across every layer you don't pin down. Between run-to-run model variance and the layers of retry logic in the system itself, "the number" was never really one number - it was a distribution I was sampling from once and calling final. Worth deciding on purpose whether that's good enough for your use case, rather than discovering it by accident three reruns in.

RedCouncil runs on Qwen Cloud (Qwen-Max) end to end - all five debate agents plus the Synthesizer. If you're building anything where multiple model calls are supposed to disagree with each other on purpose, budget real time for your evaluation code, not just your product code. Mine needed just as much debugging.