I Built an Agent That Refuses to Act - Here's Why Qwen Made It Possible

What a data team does. Automated.

When a metric drops, here's what your data engineer and analyst actually do:

• Query the data. Engineer pulls two snapshots from the warehouse - last week vs. this week, segmented by dimensions. 30 minutes.
• Build segment breakdowns. Analyst computes per-segment rates, contribution shares, significance thresholds. 2 hours.
• Evaluate significance. Is that drop on the paid channel real, or is the sample too small? 30 minutes.
• Check for confounding. Did the rate actually change, or did the population mix shift? Another 30 minutes.
• Decide. Act or escalate? 30 minutes.

Total: ~4.5 hours. $900. And no guarantee they got it right.

Here's what prove-or-abstain does with that same data:

POST /investigate { "panel": "clean" } → 2 seconds → ASSERT segment=paid (p < 0.001) → EXECUTE: webhook fires, campaign paused

Same raw data. Same conclusion. 22,500 times faster. 50,000 times cheaper.

How it knows when to act - and when to shut up

This isn't a prompt you ask nicely. The agent routes every candidate dimension (device, country, channel, plan...) through four mathematical gates. All four must pass before it's allowed to ASSERT a cause:

• Material - the drop is at least 2%. Noise isn't worth a human's time.
• Localized - one segment accounts for ≥55% of the drop. If every channel dipped equally, no single action fixes it.
• Significant - a real two-proportion z-test, p ≤ 0.01. Same perfectly concentrated drop passes on 6,000 users, fails on 60. Your analyst would do the same check - the agent just does it instantly.
• Clean - rate and mix effects aren't entangled. No confusing a composition shift with a real rate change.

Any gate fails → ABSTAIN. The agent escalates to a human with the exact failing condition named. Not "I don't know."
"Interaction share 0.63 > 0.50 - rate and mix effects are entangled; this needs a human."

This is the cost-saving mechanism. An ABSTAIN means: I just saved you from acting on a wrong diagnosis. The PM doesn't roll back a deploy. The engineer doesn't revert a feature flag. No one builds a dashboard for a phantom problem. That's where the real money is - not the investigation time, but the false action prevented.

Why Qwen

Your data team costs $200/hour. You can't just throw any LLM at this and expect it to replace them - you need one that makes reliable tool calls, never invents dimension names, and produces consistent behavior across model tiers. Qwen (via Alibaba Cloud DashScope) hit all three:

• Tool calling that doesn't hallucinate. Qwen drives the investigation by calling test_dimension(device), reading the gate result, then trying test_dimension(segment) - choosing which dimension to test, but never inventing one that doesn't exist. A determinism guard tests every dimension Qwen skipped, so a lazy LLM can never produce a false "nothing to see here." Every decision traces back to math, not model confidence.

• Model interchangeability without cost penalty. Swap qwen-turbo, qwen-plus, or qwen-max - same ASSERT/ABSTAIN verdict. Run the cheapest model, get the same correctness. The benchmark proves it across all three. At $0.40/M input tokens, an investigation averages ~4,500 tokens total - $0.004. That number isn't approximate. It's metered per request and committed in CI.

• Mock mode proves the LLM is separable. Run with QWEN_MOCK=1 - zero API calls, deterministic templates. Every verdict is bit-identical. Qwen conducts the investigation. The math decides the outcome. If you audit this agent, you can see the LLM decided nothing. That's the contract you need when you're trusting it to replace two salaries.

The one place Qwen genuinely judges?

Mapping unfamiliar CSV columns to [metric, dims..., n, c]. Ambiguous column names have no mechanically-deducible answer, so I built a two-pass self-verification system: Qwen proposes a mapping, then re-examines its own work against the sample rows. If it misclassified Temperature as a dimension instead of a covariate, it corrects itself - labeled self_verified: false, auditable and transparent.

The number that should change how you think about AI agents

I ran prove-or-abstain on 10 real public datasets - Titanic passenger manifests, US college majors with real employment stats, airline passenger counts, penguin populations, car fuel economy, the Gapminder dataset, and more. 7 out of 10 produced ABSTAIN. Because in the real world, most metric shifts are genuinely systemic. They don't localize to a clean, actionable segment. An agent that fabricates a cause on those seven is a financial liability - it triggers actions that cost more than the original investigation.

Your analyst might spend 4 hours and still reach the same conclusion: "this is systemic, escalate." The agent reaches it in 2 seconds. For $0.004. And it documents exactly why, with a replayable audit trail.

The three that did ASSERT found real, unplanted truths:

• Titanic survival localizes to sex=female (p=0.0018) - the "women and children first" effect, not the popular pclass explanation
• The STEM employment gap concentrates in majority-women college majors
• Penguin body mass variation localizes to species

What this means for your team

The agent doesn't get tired. It doesn't skip the significance check at 5pm on a Friday. It doesn't have a gut feeling that the mobile team shipped something buggy and confirmation-bias its way there. It runs the same four gates every time and names exactly which one failed. 20/20 benchmark accuracy. 0% false-ASSERT. 0% false-ABSTAIN. $0.004 per investigation. Docker. MIT license.

git clone https://github.com/Demba09/prove-or-abstain
cd prove-or-abstain && pip install -r requirements.txt
QWEN_MOCK=1 uvicorn api.app:app   # Open http://localhost:8000 - no API key needed

Built for the Qwen Cloud Hackathon (Track 4 - Autopilot Agent).