AI Coding Agents Can Pass Tests and Still Make the Wrong Decision

A question I've been thinking about after discussing AI coding agents with several developers: Is passing the test suite enough to prove that an AI agent made the correct engineering decision? I don't think it is. And this isn't just a theoretical concern. Modern coding agents are increasingly working at the repository level rather than generating isolated code snippets. OpenAI's Codex documentation, for example, describes using repository-specific AGENTS.md instructions to tell the agent how to navigate a codebase, run tests, and follow project practices. Anthropic similarly describes Claude Code searching codebases, tracing dependencies, editing multiple files, and working with CI failures. (OpenAI) That changes what "correctness" means. Consider a simple scenario A project starts with: Architecture v1 API ↓ Service ↓ Database An AI agent learns this structure and implements a new feature correctly. The tests pass. Then the architecture changes: Architecture v2 API ↓ Event Bus ↓ Services ↓ Database The same task is requested again. If the agent continues following the old architecture, its code might still: - compile, - pass existing tests, - satisfy the visible functional requirement, but still be wrong for the current system. This is the distinction I'm interested in: Code correctness ≠ Contextual correctness The Benchmark Problem Traditional coding benchmarks generally provide: Repository + Issue ↓ Agent ↓ Patch ↓ Tests / Evaluation This is valuable. SWE-bench, for example, was designed around real GitHub issues and repositories, and OpenAI created SWE-bench Verified with human validation because benchmark quality itself affects what we conclude about model capability. (OpenAI) But there is another dimension worth testing: What happens when the context changes? Recent research is already moving in this direction. SWE-ContextBench evaluates whether coding agents can reuse relevant experience across related tasks, while SWE-Explore focuses specifically on repository exploration and context retrieval rather than treating the entire coding task as a single pass/fail outcome. (arXiv) So I don't think the idea should be: "Replace existing coding benchmarks." Instead: Add controlled context-shift evaluations to them. A Practical Experiment Keep the model and task constant. Change only the relevant context. For example: Test A - Original Context Architecture: REST → Service → Database Constraint: All database access must go through Repository classes. Agent produces: Controller ↓ Service ↓ Repository ↓ Database ✅ Correct. Test B - Architecture Changed Only change: Architecture: REST → Event Bus → Service → Database Now the expected implementation should change. If the agent still produces: Controller ↓ Service ↓ Repository then we have a measurable context-adaptation failure. But There Is Another Side We shouldn't reward an agent simply for changing its answer. Suppose we change something irrelevant: README formatting The architecture hasn't changed. The agent should ideally make the same engineering decision. So a useful benchmark should test both: Relevant Context Change ↓ Decision SHOULD change and: Irrelevant Context Change ↓ Decision SHOULD remain stable This gives us two complementary properties: Context Adaptation Can the agent respond appropriately to relevant changes? Context Stability Can the agent avoid unnecessary changes when the context is irrelevant? Possible Evaluation Metrics We could measure this quantitatively. For example: Context Adaptation Rate = Correct decisions after relevant context changes / Total relevant context changes And: Context Stability = Unchanged decisions under irrelevant changes / Total irrelevant context changes Then combine these with existing measures: Agent Evaluation │ ├── Functional Correctness ├── Test Pass Rate ├── Constraint Adherence ├── Context Adaptation ├── Context Stability └── Repository Understanding I'm not suggesting this is a finished benchmark methodology. It's a direction that I think is worth experimentally validating. Why This Matters The industry is already moving toward agents that operate over entire codebases. Anthropic's recent analysis of roughly 400,000 Claude Code sessions describes agents being used for increasingly end-to-end software tasks, while engineers retain an important role in planning and directing the work. (Anthropic) As agents receive more autonomy, the evaluation question changes. For a code completion system: "Is this code correct?" may be enough. For an agent modifying a long-lived production system: "Is this the correct decision given the current state, constraints, architecture, and history of the system?" becomes much more important. The Bigger Idea Maybe the next generation of coding-agent benchmarks shouldn't only measure: Can the agent solve the task? They should also measure: Can the agent recognize when the task's surrounding reality has changed? That could give us a more realistic picture of agent reliability. Not just: Task → Code → Tests but: Task + Current Context + Constraints + Repository State + Previous Decisions ↓ Agent ↓ Decision ↓ Context-aware Evaluation And importantly, this can be tested experimentally rather than treated as a vague concept. What would you include in a context-shift benchmark first: architecture changes, security constraints, dependency changes, business requirements, or repository history? References - OpenAI - SWE-bench Verified and evaluation methodology (OpenAI) - OpenAI - Codex and repository-specific AGENTS.md context (OpenAI) - Anthropic - Claude Code and repository-level coding workflows (Anthropic) - Anthropic - empirical analysis of Claude Code usage (Anthropic) - SWE-ContextBench - context/experience reuse in coding agents (arXiv) - SWE-Explore - repository exploration and context retrieval evaluation (arXiv) Top comments (0)