Your AI Coding Agent Is Only as Good as Its Harness - What Changed in Harness Score 1.5

Introduction

Two teams can use the same model, write the same prompt, and get completely different results. The difference is often not the model. It is the repository around it. One repository gives the agent durable instructions, scoped rules, tests, linters, CI feedback, and hooks that stop dangerous actions. The other gives the agent a README and hopes for the best. That surrounding control system is the harness.

Harness Score is a free, open-source, deterministic scanner that measures how much of that system a repository actually has. It checks filesystem evidence across Cursor, Claude Code, Windsurf, Cline, Continue, Codex, Copilot, and other tools. It then returns a maturity level from L0 to L4, a score across six dimensions, and a ranked list of what to improve next. No LLM calls. No telemetry. No network access during the scan. The same repository at the same commit produces the same result.

Version 1.0 made that model stable. The 1.5 release series turns it into a broader ecosystem with customizable scoring, five documentation languages, a GitHub Marketplace Action, a public showcase, and a reproducible research repository that tests the scanner against real open-source projects.

TL;DR Since 1.0, Harness Score has added:

• A full guide in English, Brazilian Portuguese, Spanish, Simplified Chinese, and Hindi
• Repository-level configuration through .harness-score.json
• Dual maturity and effective scores for repository and local harnesses
• Custom presets and per-check rules, with protected security checks
• A published GitHub Marketplace Action
• An evidence-backed Harness Maturity Showcase
• A reproducible Harness Maturity Analysis
• Fixes for real-world hook paths, ecosystem detection, publishing, reporting, and UI problems

Try the current release: npx harness-score@1.5.1

The problem Harness Score solves

AI coding tools are increasingly good at producing code. But a capable model does not know your architecture, release process, dangerous commands, or definition of "done" unless you encode that knowledge. Teams usually add those pieces gradually:

• AGENTS.md, CLAUDE.md, or equivalent context
• Scoped rules and reusable skills
• Tests, type checking, linting, and formatting
• CI gates
• Hooks that block unsafe commands or provide immediate feedback
• Safe secret and MCP configuration

The problem is visibility. Without a measurement, teams do not know which parts exist, which are missing, or whether a pull request quietly removed a critical guardrail.

Harness Score turns those files into a deterministic diagnosis:

Maturity: L3 · Sensing
Score: 86/108
To reach L4: add a gate hook and feedback hook

It does not claim to judge whether your code is good or whether every rule is correct. It measures whether the infrastructure that guides and verifies an agent exists. That narrow scope makes the result reproducible enough for CI.

From 1.0 to 1.5: what changed

1. The guide now speaks five languages

Version 1.1 shipped the complete guide in English, Brazilian Portuguese, and Latin American Spanish. Version 1.2 added Simplified Chinese and Hindi. This was not a landing-page-only translation. The guide includes the maturity model, check catalog, multi-harness support, sensors, guardrails, and remediation recipes in all five locales. Harness engineering is a team practice. A failed check is only useful if the people responsible for fixing it can understand the reasoning behind it.

Open the multilingual guide.

2. .harness-score.json became the repository contract

Version 1.3 introduced configuration for a question that a repository-only score could not answer honestly: What does the team commit, and what does the agent on this machine actually see? The scanner now reports two views:

A repository can enable those scopes without changing the conservative CI default:

{
  "scopes": {
    "user": true,
    "system": false
  },
  "gate": "maturity"
}

Version 1.3.1 expanded user‑scope discovery across the documented global paths used by Cursor, Claude Code, Windsurf, Cline, Continue, Codex, OpenCode, Zed, and other supported tools.

3. Version 1.5 adds honest team customization

Not every check applies to every organization. For example, a company policy may prohibit repository‑local hooks. Previously, a team had to accept a permanent penalty or fork the maturity model. In 1.5, the repository can explain that decision explicitly:

{
  "extends": ["no-hooks"],
  "rules": {
    "HYG-05": "off",
    "CI-01": "error"
  }
}

The important part is how customization behaves:

• An excluded check is removed from both earned and available points. It does not grant free credit.
• Every exclusion appears in terminal, Markdown, JSON, and Action output.
• If a preset removes a dimension required for the next maturity level, the report marks that level as capped and explains why.
• Checks that detect exposed credentials - HYG-03, HYG-04, and HYG-06 - can never be disabled.

Customization is transparent, reviewable repository policy, not a hidden way to inflate a score.

Read the v1.5.0 release notes.

The GitHub Action is now on the Marketplace

The current patch, v1.5.1, marks the Action's initial publication on the GitHub Marketplace. A minimal workflow looks like this:

name: Harness maturity
on:
  pull_request:
  push:
    branches: [main]
jobs:
  harness:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: paladini/harness-score@v1
        with:
          min-level: "3"
          badge: "harness-badge.svg"

The Action can gate a minimum maturity level, write a GitHub job summary, generate Markdown and SVG badge output, and optionally maintain a sticky pull request comment with score changes. For maximum supply‑chain stability, pin the Action to a full commit SHA.

The score is now backed by public evidence

A scanner becomes more useful when people can inspect what it produces outside its own repository. The Harness Maturity Showcase is an open, evidence‑backed leaderboard. It currently combines:

• 21 pinned repositories with complete, reproducible JSON reports generated with harness-score@1.5.0
• 20 community repositories with public Harness Score badges

Numeric rankings only include full reports. Badge‑only entries display a maturity level but do not invent a numeric score. Every ranked entry links to its evidence and measured commit.

The goal is not to shame low‑scoring projects. A score describes the harness committed in a specific repository, not the competence of the people or company behind it.

The Analysis repository tests the model itself

The Harness Maturity Analysis is the research layer behind the showcase. It pins each repository to an exact commit, pins the scanner version, stores the raw reports, and generates the leaderboard and findings from those artifacts. It also compares automated results with repository‑level evidence to identify false positives, false negatives, and categories the maturity model does not represent well.

Phase 1 is complete. All 21 repositories have been rescanned with harness-score@1.5.0, and the first evidence‑backed findings are public. The blind human‑rating phase is still planned, so the project does not yet claim that every automated level agrees with human judgment. That distinction matters. Harness Score is not treating its own output as ground truth. The analysis repository exists to find where the model needs to change.

Real repositories found real bugs

The path from 1.0 to 1.5 also included less glamorous but important fixes:

• Hook path resolution: Claude Code commands such as $CLAUDE_PROJECT_DIR/.claude/hooks/setup.sh no longer fail HKS-05 when the referenced script is committed.
• Installed hook binaries: Commands under node_modules/.bin/ are recognized as installed dependencies instead of missing repository files.
• Ecosystem sensors: Maven test and Spotless configuration now count.
• Global harness paths: User‑scope scanning recognizes the real locations used by more tools, including Windsurf and Cline aliases.
• Release reliability: Package metadata and version synchronization were corrected to keep npm, JSR, the CLI, and the Action aligned.
• Reporting honesty: Configured dimensions and unreachable maturity levels now produce applicable and capped metadata instead of misleading percentages.
• Documentation UI: Navigation, dark‑theme contrast, and cross‑product links were fixed and aligned with the showcase.

Several of these fixes came directly from scanning real projects. That is the feedback loop I want for the project: scan, inspect evidence, find the mismatch, fix the deterministic rule, and run the corpus again.

Screenshot: Harness Score scanning itself

The Harness Score repository dogfoods the maturity model and currently reaches L4 · Self‑correcting. I generated the screenshot output by running this command from the analysis repository:

npx harness-score@1.5.1 D:\code\harness-score

The result:

harness-score v1.5.1
D:\code\harness-score
Maturity: L4 · Self-correcting
Score: 108/108 (100%)
scopes: repo
Detected: Antigravity, Claude Code, Cline, Codex, Cursor, Windsurf

Context & Guides ████████████████████ 100%   20/20 pts
Skills & Commands ████████████████████ 100%   17/17 pts
Hooks & Guardrails ████████████████████ 100%   14/14 pts
Sensors & Feedback ████████████████████ 100%   20/20 pts
CI Feedback        ████████████████████ 100%   14/14 pts
Hygiene & Safety   ████████████████████ 100%   23/23 pts

All checks passed · this repository is fully harnessed.

Next stop: Product Hunt

I am planning to launch Harness Score on Product Hunt next week. The Product Hunt launch is not the finish line. It is a way to put the tool in front of more teams, collect more real repository evidence, and discover where the maturity model is still wrong or incomplete. If you test it before the launch, the most valuable feedback is concrete: share the repository, check ID, file path, and why the result is a false positive or false negative.

Try it and share your result:

npx harness-score@1.5.1

Then explore the ecosystem:

• Harness Score repository
• Documentation
• GitHub Marketplace Action
• Harness Maturity Showcase
• Harness Maturity Analysis
• v1.5.1 release notes

What level did your repository get, and which failed check surprised you most? Drop both in the comments.