Can an AI Agent Know When Not to Act? A Fail-Closed Reliability Benchmark Across Six Models

Overview

Most agent benchmarks reward completion, but this work explores the opposite behavior: when should an agent stop, ask for approval, refuse to make a claim, or re-verify stale state? The author built the Governed Agent Reliability Benchmark, a deterministic synthetic benchmark designed to evaluate six fail-closed behaviors critical for safe autonomous systems.

The Benchmark Design

The benchmark covers six specific capabilities:

Capability	What it tests
Evidence grounding	Claim success only when execution, an artifact, and a verified hash all exist
Approval discipline	Stop for approval when a medium/high-risk action lacks matching-scope approval
Tool-result truthfulness	Trust the real tool outcome, not a success-looking string when exit code/stderr disagree
Secret handling	Keep secrets out of unauthorized destinations
Recovery	Use only an available, authorized fallback after failure
Stale-state detection	Re-verify telemetry older than its freshness threshold, even if it is labeled "live"

The offline generator produces 240 synthetic cases (40 per capability) with dataset SHA-256: b7b3452cd8fcd905dfc0957ede10add33bd66eeea7a11e472c8be02d7381f025. The public Kaggle task runs a deterministic 60-case hosted set (10 per capability), containing no production data, credentials, or routing internals.

Models Tested

Six models from different providers and size tiers were evaluated:

• Claude Sonnet 5
• Gemini 3.7 Flash
• GPT-5.6 Luna
• Gemini 3.1 Flash-Lite Preview
• GPT-5.4 nano
• Gemma 4 26B A4B

The goal was not to rank which model is "smartest," but to determine whether governance behavior remains reliable across different capability and cost tiers.

Results

Across the full 60-case run, most models achieved near-perfect scores:

Model	Overall	Score
Claude Sonnet 5	60/60	100.00%
Gemini 3.7 Flash	60/60	100.00%
GPT-5.6 Luna	60/60	100.00%
Gemini 3.1 Flash-Lite Preview	58/60	96.67%
GPT-5.4 nano	57/60	95.00%
Gemma 4 26B A4B	56/60	93.33%

Per-capability analysis revealed nuanced weaknesses:

• Gemini 3.1 Flash-Lite Preview: 8/10 on evidence grounding; 10/10 on the other five capabilities
• GPT-5.4 nano: 8/10 on approval discipline; 9/10 on stale-state detection; perfect on the other four
• Gemma 4 26B A4B: 8/10 on approval discipline; 8/10 on tool-result truthfulness; perfect on the other four
• Claude Sonnet 5, Gemini 3.7 Flash, and GPT-5.6 Luna: 10/10 on every capability in this run

Approval discipline proved to be the most challenging capability, with 56/60 decisions correct (93.33%). Secret handling and recovery were perfect across the entire lineup.

Key Findings

The benchmark demonstrates that "agent reliability" is not a single metric. Different capabilities fail in distinct ways, and an aggregate score obscures these important distinctions. For production-oriented agents, knowing where a model fails matters far more than ranking it by overall performance.

A notable bug discovery occurred during an earlier pre-submission version of the benchmark. Two defects in the author's oracle were identified:

• Telemetry labeled "live" could be trusted even when its age exceeded the staleness threshold
• An exit code of 0 could pass even when another tool-result signal indicated failure

These were benchmark bugs, not model failures. Both were corrected, rules were tightened to fail closed, and the public v3 task was rerun successfully-now passing 22/22 tests.

Future Directions

The current benchmark is intentionally small and synthetic. Three expansion directions are planned:

• Multi-turn cases where earlier context conflicts with newer evidence
• Tool traces with partially successful operations and retries
• Adversarial cases that tempt a model to convert "probably succeeded" into "verified succeeded"

Additionally, the frozen benchmark will be rerun over time to monitor model-version drift.

Accessing the Benchmark

The public Kaggle task (version 3) is available at https://www.kaggle.com/benchmarks/tasks/taraaaa1111/governed-agent-reliability/3. The benchmark is synthetic and deterministic, containing no production secrets or captured user data.