I Built Scenario Packs for Agent Regression Testing. The Integration, Not the Judge, Broke Me.

I thought the hard part would be the scoring. Write clean YAML. Define expected behavior. Run agents. Compare scores. Catch regressions. Ship with confidence. That mental model lasted about one afternoon of real-agent field testing. The thing that broke was not the judge. It was not the rubric design. It was the realization that a scenario pack is only as honest as the path between your harness and a real, messy, third-party agent that imports ffmpeg at module scope, hardcodes gpt-3.5-turbo , and writes to /root the moment you touch it. This is the second article in a series about EvalForge, an OSS evaluation harness for tool-using AI agents. Article 1 made the case that agent evaluation is a different problem from model evaluation because the path matters, not just the answer. The launch article is the longer story of what real agents taught me once the code was public. This one is narrower: scenario packs, baselines, and scoring. The three concrete things I built, and the one that broke first. The Scenario Pack Is a Contract, Not a Test File Before I get to what broke, I need to show what I actually built, because the design decisions in the pack format are where the engineering lives. This is a scenario from the launch pack. One of twenty. It looks clean. It is clean: # scenarios/core-launch.yaml - launch-01-account-policy # https://github.com/deghosal-2026/agent-eval-forge/blob/main/scenarios/core-launch.yaml - id: "launch-01-account-policy" title: "Account policy lookup" goal: "Retrieve a specific policy detail using one tool call" input: "What is the return policy for premium customers?" allowed_tools: - name: "policy_lookup" disallowed_tools: [] expected: type: exact value: "Premium customers receive a 60-day return window with free return shipping." metrics: task_completion: {threshold: 1.0} output_correctness: {threshold: 0.8} tool_correctness: {threshold: 1.0} step_efficiency: {threshold: 0.7} tags: [retrieval, single-tool] difficulty: easy budget: {max_steps: 3, max_tokens: 300} Twenty scenarios shipped in v0.1 across ten families: single-tool retrieval, multi-tool synthesis, structured extraction, tool argument precision, refusal, ambiguity clarification, budget constraints, failure recovery, coding-agent regression, and classification. Eight more for security: prompt injection, exfiltration, SSRF, sandbox escape. The architecture is straightforward: CLI runs the pack through a core runner. Runner delegates to an adapter. Adapter talks to the agent. Scorer evaluates the trajectory. Judge fills in semantic gaps. Diff engine compares the result against a saved baseline. The ground-truth boundary The most important design decision in the pack format is not visible in the YAML. It is what the agent never sees. The expected and metrics fields are evaluation-only. They are stripped before the agent receives anything. The build_invocation_payload function in the adapter base is the enforcement point: # src/evalforge/adapters/base.py - build_invocation_payload def build_invocation_payload(scenario: Scenario, run_id: str) -> dict[str, Any]: return { "schema_version": "evalforge.invocation_payload.v1", "run_id": run_id, "scenario_id": scenario.id, "input": scenario.input, "context": scenario.context, "allowed_tools": [tool.model_dump() for tool in scenario.allowed_tools], "disallowed_tools": [tool.model_dump() for tool in scenario.disallowed_tools], "budget": scenario.budget.model_dump() if scenario.budget else {}, } Notice what is not in that dict. No expected . No metrics . No threshold . No goal . The agent gets the input, the tool surface, and a budget. It does not get the answer key. It cannot game what it cannot see. This is not a convenience - it is a correctness boundary. If ground truth leaks into the agent's context, every score is suspect. The Scenario model documents this in its docstring: "expected /metrics are evaluation-only and never sent to agents." The Baseline model and the ComparisonEngine both depend on that boundary holding. If it breaks, the regression story breaks with it. There is a second boundary in the same file, and it is the kind of thing nobody talks about until it bites them. The _sanitize_agent function strips API keys and tokens from adapter config before writing them into run artifacts. Pass api_key in your adapter config - it never reaches the artifact store. Secrets do not persist. I would call this a feature, except that calling it a feature implies it is optional. It is not. A third one: when the adapter parses agent output, a "completed" run that produced no output at all is treated as an error, not a pass. The comment in _artifact_from_envelope is blunt: "blank completions usually signal a dead entry point or empty tool result, and must never count as passes." A blank completion is a failure wearing a pass costume. The harness refuses to count it. These three boundaries - ground-truth stripping, secret sanitization, blank-completion rejection - are the ones I would fight to keep if I had to rebuild from scratch. Everything else is negotiable. These are not. If you are building an eval harness, I want to know: where is your ground-truth boundary? Is it enforced at a single function, or is it a convention that depends on every adapter remembering to do the right thing? The Adapter Problem Started Before Scoring Even Ran Then I sourced 19 OSS agents from GitHub - 11 LangGraph, 8 PydanticAI - using a star-bucket strategy. High-star repos for maturity signals, medium for real-world mess, low to see if the tool adds any signal in chaotic codebases. The sourcing methodology is documented in docs/hard-won-lessons.md. Nine passes. Out of 95 scenario-agent combinations. Not nine-per-agent. Nine total. The instinct when you see nine passes is to blame the judge. Switch from gpt-4o-mini to gpt-4o. Tune the rubrics. Add more scoring dimensions. I ran the same passes on two judge tiers - gpt-4o-mini (cheap) and gpt-4o (better). Same outcome both times. Nine passes. The better judge did not surface a single regression or improvement the cheaper one missed. The bottleneck was not the scoring layer at all. The bottleneck was whether the harness could run the agent in the first place. Five ways real agents broke the adapter I documented these in the hard-won lessons file, but these are the patterns that actually hit: Absolute writes at import time. Several agents wrote to /root/something inside their init.py . The harness runs in a locked-down sandbox. Import failed before any evaluation code executed. The fix was not elegant: redirect HOME , TMPDIR , and XDG_CACHE_HOME to per-agent .cache directories. Agents that still wrote to absolute paths got quarantined. Gateway-bound imports. Multiple agents did ChatOpenAI(api_key=os.getenv("OPENAI_API_KEY")) at module scope. If the key is missing, the module itself raises. You cannot import it. You cannot evaluate it. The workaround was dummy env vars for the local tier. Agents that required real gateway connectivity got quarantined for local runs. Hardcoded model names. ChatOpenAI(model="gpt-3.5-turbo") at module scope. I pointed OPENAI_BASE_URL at a local OMLX server running Qwen3.5-9B-MLX-4bit . The agent still asked for gpt-3.5-turbo . OMLX does not serve that model. 404. The fix was monkeypatching ChatOpenAI.init before the agent module is imported - and I learned the hard way that Pydantic v2 field-default patching does not work for this. It has to be init . It has to run before import. Typed StateGraph with no chat surface. Some LangGraph agents use typed StateGraph with internal domain state fields. The harness sends chat messages. The agent expects AgentState with typed keys. There is no bridge. I had to write thin evalforge_wrapper.py modules per agent to translate. This is not a harness bug. It is a design gap: the harness assumes a message surface, and typed-graph agents do not expose one. Database bootstrap at import. create_async_engine(DATABASE_URL) and FAISS.load_local(...) inside module scope. The harness should not be patching around an agent's entire infrastructure bootstrap. I learned to classify agents by import-time side effects - no infra, needs DB/keys/files, needs running server - and skip the ones I could not run locally. Move on. Do not fight databases. The lesson I walked away with: a scenario pack tests your adapter before it tests your agent. If the harness cannot faithfully run a random third-party agent, the signal you are measuring is integration friction, not agent quality. Friction is real and worth measuring. It is just not the same thing, and calling it the same thing is how teams ship agents they do not actually understand. What I would build differently The harness currently classifies agents into three tiers - local, Docker, quarantined - and moves on. That triage works for a first pass but paper-bags a real architectural choice. Right now the default adapter imports agent code directly into the harness process. The python_import adapter shoulders the import, and the isolated adapter wraps it in a subprocess for some safety. But the boundary is still "shared Python process" at heart. A cleaner design would be: the harness never imports agent code. It always communicates through a strict stdin/stdout contract. The subprocess adapter already exists and already works this way. Every agent gets a well-defined protocol: invoke(input, tools, budget) → trajectory . The harness does not care what language the agent is written in, what it imports, or what it writes to disk. The import-based adapters were faster to wire for the first 19 agents. I would build the subprocess boundary as the one true path from the start, and treat import-based adapters as an opt-in optimization for agents you already trust in-process. This is the one I keep turning over: should an eval harness ever share a process with the thing it is evaluating? Or is process isolation the minimum bar for honest measurement? I lean towar