How do you unit test an agent skill?

Agent skills are prompts, not code, and there’s no compiler to catch a broken one. Agent skills ship on the honour system. You rewrite one, run it twice, post something convincing in Slack, and that’s the review. Is it faster? More reliable? Going to cost more? This isn’t a strategy. This is astrology for prompts. That bugged me. Not because I thought people didn’t know what they were talking about, but because they couldn’t prove they did. I can’t ship a colour change on a button without a design review and two approvals. Greg can push a new database-migration skill on a Friday afternoon because he felt good about it after a couple of coffees. “Trust me,” he says. “It makes way fewer mistakes.” So I built tests for them. skilleval takes a SKILL.md, a prompt, maybe a fixture for the agent to chew on, and runs a real agent. Then you assert on what actually happened: tools, files, cost, whether the skill even switched on, the final message. Pass or fail. No second model grading the first. I’m not smart enough to debug two LLMs arguing. expect(result).costUSD.toBeLessThanOrEqual(2) expect(result).skills.activated.toInclude("refactor-code") expect(result).toolsUsed.toInclude("read", "edit").not.toInclude("web") expect(result).toolCalls.toIncludeInOrder([ { name: "edit", args: { path: "src/foo.ts" } }, { name: "shell", args: { command: /git commit/ }, exitCode: 0 }, ]) expect(result).toolCalls.not.toIncludeInOrder([ { name: "shell", args: { command: /rm -rf/ }, exitCode: 0 }, ]) expect(result, workspace).file("src/foo.ts").toHaveBeenModified().toContain(/function Foo/) expect(result).finalMessage.toMatch(/foo.ts refactored/) Results get saved as artefacts, so if you edit the skill and re-run you can see the delta. The skill won’t do the same thing twice, which is why the test is on what it left behind. If you’re paranoid, run it N times and pick a pass rate. This will not stop Greg. But “it makes way fewer mistakes” now has a result.json behind it, or it doesn’t. Same bar as the button. That’s all I wanted. This is one way to test skills. I’m sure there are others, and I’d love to know what people are already doing. Repo’s here, if you try it, let me know your thoughts!. Top comments (0)