I built a coding agent in ~970 lines of Python and benchmarked it honestly

The thesis

TL;DR: I built nano-harness a coding agent in ~970 non-blank lines of Python (5 files, 3 tools, 2 providers, MIT). It scored 59.6% (53/89) on the full Terminal-Bench 2.0 suite with Claude Opus 4.8. Along the way an independent frontier model (GPT Sol 5.6) reviewed the code and gave it a 4/10, which turned out to be the most useful thing that happened to the project. This is the whole arc: the score, the bugs, and the parts that went wrong.

Why I built a harness at all?

Back in February I started reading about harnesses from none other than.... Andrej Karpathy, I think his llm-wiki had just gone viral and I saw this repo. So I read it and started reading more on harnesses and loops. This was before EVERYONE was talking about loops and harnesses and every YT video was about prompts and so on. I wanted to learn something new, and at the time was training my own model using Azure Machine Learning (which costed an arm and a leg to learn), then was using PyTorch to learn patterns within my AI Operating System.

Anyways, most agent projects I was following led with features: plugins, orchestration graphs, UI dashboards, memory systems (their obsidian "second brain") - all the same thing. I had a harder time finding small, readable harnesses paired with a reproducible full-suite benchmark score (and if you know anything about wanting to benchmark your harness it IS NOT CHEAP!). This really stuck with me and I didn't know why more people were talking about this. To me this was where everything was heading, and the harness is the part of an agent you can actually engineer, so it's the part you should be able to code and measure.

So the thesis of nano-harness is score-per-line-of-code: the smallest readable harness that still puts up a real number on a real benchmark. Think of Karpathy's nanochat, but for agent harnesses - small enough to read end-to-end in one sitting, honest enough that the number means something. My original goal was ~500 lines of code and to get a real coding benchmark above 45% and keep progressing. This was going to be a challenge for me at least.

What ~970 lines buys you

The whole harness is one loop and three tools:

• bash - a persistent shell (cwd, env, and state survive between calls)
• read_file - line-numbered reads with slicing
• edit_file - exact unique-match string replacement

That's it. No embeddings, no planner, no sub-agents. bash subsumes ls, grep, build, test, and install. The model does the coding; the harness has exactly one job: keep the run alive and keep it honest.

"Alive" means: retry transient API failures, truncate history before the context window dies, nudge a continuation when output gets cut mid-tool-call, kill and respawn the shell on a hang, and turn every exception into a structured result instead of a crash.

"Honest" means: a failing command must read as a failure, and "done" must be earned. The load-bearing piece is a verify gate. Once a run has used tools, the first "done" is challenged: re-read the task, run the relevant checks, prove it. A later completion is accepted only when successful tool evidence has appeared since that challenge. If the pushback or iteration budget runs out without that evidence, the result is returned as "unverified" and therefore is not a success. Tool-free tasks (a pure question) are allowed to finish normally; there's nothing to verify.

The benchmark arc: 20% → 80% → 53.9% → 59.6%

I benchmarked on Terminal-Bench 2.0 via Harbor - 89 tasks, each in its own Docker container, with the exact shipping harness installed per container. My progression, in order:

The final run: 16.5 hours, two tasks in parallel, errored trials counted as failures (10 wall-clock timeouts on the heaviest tasks plus one container OOM - all counted against me).

Public Terminal-Bench results are agent-model pairs, so model quality and harness quality are entangled in every number. The official 2.0 board is topped by things like Codex CLI + GPT-5.5 (82.2%) and WOZCODE + Opus 4.7 (80.2%). I couldn't find a verified entry for my exact model on that table, so I'm not going to dress 59.6% up as a clean measurement of "the harness gap." It's my self-run result under the conditions disclosed here - a ~970-line harness against a suite where the tuned, much larger harnesses live in the low 80s - with the code and the task-level record in the repo for anyone who wants to check.

Honestly I was proud of all the work I had put in over 3.5 months, while having my full time job as SR Engineer, and hosting a API with active paying users updating daily, and running a free web app with 100 active users and no marketing on anything besides a couple reddit posts. Staying up till 2/3AM every night, waking up at 5AM... Which isn't healthy and I actually had a health scare due to lack of sleep and all the sitting I was doing despite working out every day. But I think we can all admit that not only is learning nowadays easier and much more fun, but with AI if you miss a day you feel like you were gone for a month - and it can be overwhelming when there are weeks like these past few where it's model after model, benchmark after benchmark. And on top of that I am working with what I got; I don't have a mac-mini or NVIDIA SPARK. I bought the Beelink SER 10 MAX OpenClaw edition 64GB because it was the only 64GB available in my price range. And oh man do I wish I had a nice mini PC to run local models and test what I want.

The part where GPT Sol gave my code a 4/10

Here's the part I actually want to tell you about. And I just want everyone to understand that these benchmarks these frontier models score - the harnesses they have are thousands, even tens of thousands of lines of code...

After the first full benchmark run, I pasted the five core files into GPT Sol because it had just come out while I was 13 hours in to my first run. I used Opus 4.8 to wrap my harness around so it was a "competitor's", no shared context, and I asked for an adversarial/constructive code review. It scored the code 4/10 and produced a list of findings.

The most consequential one was also the best catch: A shell command that failed was reported to the model as success. My bash tool captured output but not exit status. false, a failing test suite, grep with no match - all of it came back looking clean. And I felt like an idiot (...must of been the nights I woke up with head in the middle of my keychron... also not happy about that). Which means my verify gate - the whole "keep it honest" centerpiece - could be satisfied by a failing test run. The harness's whole reason to exist had a hole in the middle of it.

So I worked through the findings test-first: every fix landed with a failing regression test before the fix. (Since this is an article about honesty: nano-harness was built with heavy AI-assisted coding, as I'm sure you figured by now. But unlike most, I made sure I learned every step. I directed the work, made the calls, and ran every benchmark and review, but I did not hand-type all 967 lines. I'd say honestly I wrote ~100. The review gauntlet below used three independent review passes with no shared conversational context, which helped surface different classes of failures.)

Then I had a second independent review pass over the result. It scored 6/10 and caught two of my fixes as only half-fixes: I'd stored tool calls in two places, and my context-truncation pass only shrank one of them - so the OpenAI serialization path silently re-inflated giant tool arguments I thought I had truncated (and I know I did, but I must have not committed that and let it slip by me). My CRLF fix preserved line endings for uniform files but homogenized mixed-ending files. Fixed those too.

Then a third pass - a multi-agent cloud review fleet - found exactly two nits (a missed retry case for client-side API timeouts, and an explicit JSON null argument bypassing a default). Fixed both.

Total damage across the gauntlet: ~20 real bugs, test suite grown from 52 to 87 tests. Among the fixes:

• Nonzero shell status now raises a tool error, so a plain failing test command no longer counts as successful verification evidence
• The shell's sentinel protocol survives set -x (a substring match used to latch onto the trace line and mis-frame every subsequent command)
• edit_file preserves file permissions (it used to silently strip the executable bit off any script it edited on Linux benchmark containers - that's a task-killer), edits through symlinks instead of replacing them, and leaves untouched lines' endings byte-identical in mixed CRLF/LF files
• Process-tree kills are verified and reaped, no zombie shells accumulating across timeouts
• The verify gate fails closed: out of pushbacks means "unverified", not fake success

And here's the punchline: the benchmark handed me - the pre-hardening harness scored 53.9%. The post-hardening run - same model, same suite - passed five more tasks for 59.6% - a 5.7-point gain, but also an extra ~180 lines of code. These are stochastic single runs, so I can't prove every point came from a specific fix. What I can say: the intervening changes were correctness and safety fixes, not task-specific benchmark patches, and the next full run scored higher. The biggest single fix was making failure look like failure to the model - and it turns out models get further when their harness stops lying to them... who would have thought...

What this cost

More than I can quote to the exact dollar, and the honest reason is a gap in my own adapter: nano logs token usage to the agent's stdout instead of reporting it back into Harbor's result schema, so the run's summary JSON has null token fields. Adding up the surviving per-task agent logs gives about 3.0M tokens (≈1.54M in, ≈1.43M out) - and since a later retry pass overwrote a few of those logs, treat it as an estimate, not a receipt.

At Opus 4.8's July 2026 list rates ($5/M input, $25/M output), that's roughly $45 for the full 89-task run. A 10-task iteration slice runs one to two hours, on the order of $4-$6. Cheap enough that the bottleneck is wall-clock, not budget.

Clone the repo

The repo is github.com/TroyJLorents-GH/nano-harness. MIT, ~970 non-blank lines across agent.py, tools.py, providers.py, prompts.py, cli.py, with the 87-test suite, the Terminal-Bench adapter, the full benchmark breakdown, and the re-review prompt and resulting fix history in the tree.

If you take one thing from this: the harness's job is not to be smart - the model is smart. The harness's job is to keep the run alive and refuse to let anyone lie, including the model, and including you. The 4/10 review was the moment that thesis got real. Publishing the whole arc - the bad score, the half-fixes, the variance - is the point. If you review the code and find bug #21, open an issue. That's the game.