How to Know If Your Claude SKILL.md Actually Works

I spent today shipping a tool I've wanted for months. If you build with Claude, you've probably written a SKILL.md file. And you've probably shipped it based on gut feel. That changes today.

The problem nobody talks about

Skills are just system prompt injections. The honest question is: does this skill actually improve Claude's outputs, or does it just feel like it does? Most teams answer this by eyeballing a few responses. That's not evaluation. That's vibes.

Three things make vibes-based skill evaluation dangerous:

• Position bias - if you ask Claude to compare its own outputs, it favors whichever it sees first
• Silent regression - model updates, skill edits, and context changes can silently make a skill worse
• No shared rubric - every engineer scores skills differently, so "this skill is good" means nothing

What I built

skilleval - a CLI that gives you a repeatable, objective score for any SKILL.md in under 2 minutes.

npx @dileeppandiya/skilleval ./my-skill --tasks ./tasks.yaml

Real output from the sample skill in the repo:

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
skilleval results - api-design - 2 tasks
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Skill effectiveness: +0.3 / 3
Tasks improved: 1 / 2 (50%)
Tasks hurt: 1 / 2 (50%)
Confidence: UNRATED (use --runs 3+ for confidence)

task-003  +2.5  Output A provides more robust API design...
task-004  -2.0  Output A is more comprehensive...
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Runner: claude-sonnet-4-6 | Judge: gemini-3.5-flash
Estimated API cost this run: $0.101
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Notice the mixed signal. The skill helped on task-003 but hurt on task-004. skilleval doesn't inflate scores to make skills look good. It reports what the judge actually found.

How it works

• Blind A/B testing - each task runs twice concurrently, with the skill injected into the system prompt vs. raw context only.
• Randomized judge - a Gemini Flash judge compares outputs. Which output gets labeled A or B is randomized per task with a seeded RNG, eliminating position bias completely.
• Margin-based scoring - the judge returns a winner + margin (0–3): margin 3 gives 3.0/0.0, margin 0 gives a genuine tie at 1.5/1.5.
• Honest confidence - single runs show UNRATED. One sample tells you nothing about stability. Real confidence (HIGH/MEDIUM/LOW) only appears at --runs 3+.

skilleval ./my-skill --tasks ./tasks.yaml --runs 3

Five things that make it different

Deterministic assertions - not everything should be left to LLM opinion:

tasks:
  - id: login-endpoint
    prompt: "Design a login endpoint"
    assertions:
      must_contain:
        - "POST"
        - "401"
      must_not_contain:
        - "GET /login"
      min_length: 100

Assertion failures automatically count as hurt tasks, no LLM needed to know "missing POST method" is wrong.

Multi-turn conversation tasks - most real skills operate across turns, not single prompts. The skill injects into the system prompt for the full conversation, and the judge sees complete context when scoring.

Run history + regression detection - every run auto-saves to .skilleval/history/. After two runs:

skilleval diff ./my-skill

── skilleval diff: api-design ──────────────────
vs previous run: 2026-07-11T14:30:00Z
Effectiveness: +0.3 → +0.8 (+0.5 ↑)
Tasks improved: 1 → 2 (+1 ↑)
Tasks hurt: 1 → 0 (-1 ↓)

This is "skill hell" prevention in practice - you can see the exact moment a skill started regressing.

Skill version comparison - test v1 vs v2 directly:

skilleval ./skill-v1 --compare ./skill-v2 --tasks ./tasks.yaml

No more "I think v2 is better." Now you know.

One-line CI integration - block PRs that silently break skills:

on:
  pull_request:
    paths:
      - '**/SKILL.md'
jobs:
  skilleval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: dileepkpandiya/skilleval@main
        with:
          skill-path: ./my-skill
          tasks: ./tasks/tasks.yaml
          fail-below: '0.3'
          fail-if-hurt-pct: '50'
          anthropic-api-key: ${{ secrets.ANTHROPIC_API_KEY }}
          gemini-api-key: ${{ secrets.GEMINI_API_KEY }}

Exit code 0 = pass, 1 = gate failed, 2 = error.

Cost

Use --cost to see an estimate before spending anything. Gemini Flash is the default judge, and the free tier handles casual iteration easily.

Quick start

Try it immediately on the built-in sample:

git clone https://github.com/dileepkpandiya/skilleval
cd skilleval
npx @dileeppandiya/skilleval ./samples/api-design \
  --tasks ./tasks/sample-tasks.yaml

Scaffold a new skill:

skilleval --init ./my-new-skill

Install globally:

npm install -g @dileeppandiya/skilleval

You'll need ANTHROPIC_API_KEY for the Claude runner and GEMINI_API_KEY for the default judge.

What's still missing

Honest gaps in v0.3.0:

• Tool-call evaluation - if your skill affects which tools Claude calls, text-output scoring misses that
• Visual history dashboard - the diff command is CLI only, no charts yet
• Local model judge support - no Ollama/local-model judging for fully offline eval yet

The repo is MIT licensed, open source, TypeScript. 38 unit tests, zero API calls needed to run the test suite, GitHub Action included.

👉 github.com/dileepkpandiya/skilleval

What are you using to evaluate your skills today? I'd genuinely love to know what's broken about this for your use case - you can file an issue or drop a comment below.