Your SKILL.md is production config. Test it like one.

Your SKILL.md is production config. Test it like one. You maintain agent skills - the SKILL.md instruction files that steer Claude Code, Cursor, Codex, or any coding agent. Careless edits feel cheap: rewrite one instruction line, done. The problem is what happens next. That one line silently changes how the agent behaves on the next run. Files you didn't ask it to touch. A tool call sequence that skips a step. A task that used to complete in one shot now stumbles through four attempts. Usually you find out weeks later. Not from a test - from a user. Text diffs can't tell you what an agent will do differently A git diff of SKILL.md shows you which words changed. It can't show you what the agent will do differently, because agent behavior is emergent. The same prompt line can produce completely different tool trajectories depending on context. Prompt evals miss it too Offline evals score a final answer against a gold label. They don't see the trajectory: the extra tool hop, the silent regression where the agent still prints the right string but took a forbidden shortcut to get there. Here's a real example. I dogfooded skilldiff on itself - it tests its own skills. Two versions of a note-taking skill were run against the same scenario: old skill: 2 tool calls, 1 file(s) changed, 0 command(s) run files: NOTES.md new skill: 3 tool calls, 2 file(s) changed, 0 command(s) run files: NOTES.md, TODO.md Assertions: ✓ [files_changed] NOTES.md ✓ [tool_calls] read ✓ [tool_calls] write ✗ [must_not] files_changed does not include TODO.md actual (new): VIOLATED - TODO.md was changed note: this is a REGRESSION - old skill passed, new skill fails Both versions appended SPIKE RAN OK to NOTES.md - the standard diff would look fine. But the new version also created TODO.md, which the scenario explicitly forbids. A text diff would never have caught that. That's behavioral regression. The approach: run the skill twice, diff what the agent actually did skilldiff runs your skill in a real agent harness against a fixture repo - twice: - Old - the skill as it exists on the base branch - New - the skill as changed in your PR Then it captures what the agent actually did: files changed, commands run, tool calls made. And it asserts on those observations. Five assertions you can write in plain YAML | Kind | Meaning | |---|---| files_changed | paths the agent modified | commands_run | commands the agent executed | tool_calls | tools invoked (normalized across harnesses) | must_not | forbidden files / commands / tools | output_contains | substrings in the final output | A scenario looks like this: skill: notes-helper fixture: repo/notes-helper-scratch expect: files_changed: [NOTES.md] tool_calls: [read, write] must_not: files_changed: [TODO.md] Drop it into your PR flow npx skilldiff init # discovers .claude/skills, skills/, .agents/skills # and writes a starter scenario per skill npx skilldiff run skilldiff/notes-helper.scenario.yaml --live --base origin/main For CI it replays recorded traces, so it's deterministic and needs no API key or credits. The PR that touched a skill gets a comment with the behavior report; failures gate the merge. Live runs use whatever harness and login you already have. There's no shared API key shipped with the tool - each contributor runs on their own account (opencode, Claude Code CLI, Cursor, Codex, Freebuff). Honest limitations (v0.1) Skills are emergent, so a single run is a sample. Recorded/CI mode is deterministic; live runs vary run to run. Five assertion kinds is deliberately few. They cover the 80% case: files, commands, tools, forbidden behavior, output markers. This is not a replacement for full eval harnesses like Inspect or Promptfoo. It answers one question: did this change regress agent behavior I'm relying on? Contribute If your harness is missing, an adapter is roughly 40 lines - and the highest-value way to contribute. Repo: https://github.com/scs0209/skilldiff If you maintain skills and this doesn't match your workflow, I'd genuinely like to hear how you review skill changes today. Top comments (1) The "single run is a sample" line is the part worth sitting with. Recorded/CI mode is deterministic, but that's exactly the mode that won't catch a regression only showing up on the live, non-deterministic path. On the review-workflow question - closest thing here to what you're describing is a numeric gate on drafted output instead of trajectory diffing. Score every draft against target features, warn below a threshold, flip warn to block once new output actually holds the line. Same instinct as skilldiff though - don't trust a read-through, make the actual output pass a check before it ships.