DEV Community

Your agent run passed. Can you prove it was allowed?

Overview

Most agent teams can show a run. Few can show, in one re‑runnable command, whether that run was allowed under a policy. The audit introduced here answers the latter question by checking cost and call caps, the declared artifact contract, and forbidden markers, then emitting a canonical audit_hash.

Policy‑as‑Code Audit

The audit is deliberately boring:

agent-lab-trust audit --policy policy.json <run-root>

A policy can declare:

  • max_cost_usd and max_calls per run
  • required_artifacts plus required_artifacts_mode: all | any
  • forbidden_markers

The output is a finding list and a canonical audit_hash.
deletion-proof --output <path> writes the deletion evidence used in the same policy flow.

Demonstration Results

The audit was run over 13 archived GenMentor runs.

  • Trust‑layer default contract (output/structured.json or results/structured.json):

    • 0/13 passed
    • missing_artifact ×13
    • cost_exceeded ×1
  • Declared GenMentor contract (archive.json and summary.json, mode all, cap 5.00):

    • 13/13 passed
    • no findings

The data did not change. The contract did. Dashboards answer “what happened.” Governance needs “what was allowed, and by which rule.” A run can succeed and still violate a budget, write an unexpected artifact, or carry a marker that should never ship.

Audit Hashes

  • Default structured contract: 2c122c4ff2c0d5eed11a0fc23b4ff717002864dbd941402e8fe18d6d17c96ce6
  • GenMentor contract: 368b75a06c0f5c205436d287881cfddcbabd94b7ad34a860e852fba59dad7ab5

Cost Finding Details

The single cost finding is real:

  • replay-8of8-20260920-a recorded 3.9529266 under the trust‑layer cap 0.60.
  • Under the GenMentor policy cap 5.00, it passes.

Caps are part of the contract too.

Lesson Learned

The lesson is not “the audit is noisy.” It is that an audit without an explicit run‑family contract silently embeds one family’s format. The policy has to name the contract, the caps, and the markers.

Reproduction Instructions

Reproduce the audit with either:

docker run --rm ghcr.io/janzong/agent-lab-trust:rc2
# or
bash scripts/reproduce.sh

Expected results:

  • 13 passed under both TZ=UTC and TZ=Asia/Shanghai
  • report_hash a841b192981fd7e7
  • deletion audit_hash 4f0193abbd49a0f9

The :rc2 tag points at the latest published build; SERIES.md pins the verification digest sha256:2e178e63fff30e70ac68501cb17e5316097875932d1d7085e669ce4f8d5a105f.

Limitations and Next Steps

What this does not prove:

  • No real governance deployment
  • No external reproduction yet
  • No real decision changed yet
  • The 13 runs are a private synthetic family, not published data

Series and Links

This is piece 3 of a three‑part line:

  • agent-charters: what people actually tell agents.
  • agent-lab-trust: whether an agent run is valid and reproducible.
  • this audit: whether a run was allowed under a declared policy.

Index: https://github.com/janzong/agent-lab-trust/blob/main/SERIES.md

Call: Run the audit on a run family you own. Tell me where the policy and the artifacts disagree. I am also looking for 3 independent reproductions of the trust layer: https://github.com/janzong/agent-lab-trust/issues/1

Read on DEV Community ↗ ← Back to News

Comments

No comments yet. Start the discussion.