Learning AI orchestration and harness engineering by building an autonomous engineer for a bank

I took on this project not knowing much about AI orchestration or harness engineering. A few months in, they are the two things I think about most. This is what I learned, written for whoever is about to fall down the same rabbit hole.

The project is an autonomous agent at Eko that resolves real support work against a live financial system, end to end, with no human clicking go. I had assumed the hard part would be the model, the prompting, the cleverness. It is not. The model is the easy part. Everything around it is the job, and that everything has a name: the harness.

What harness engineering actually is

A harness is all the scaffolding that turns a smart but unreliable model into a system you can trust to run on its own. It is the orchestration loop that decides what happens and in what order. It is the retries and the recovery when a step fails. It is the deterministic code that holds the authority so the model never has to. It is the evaluation that tells you whether a change made things better or worse. None of that is model work. All of it is what separates a demo from something that runs in production against real money.

Before this I had helped build an agent on a lexical architecture with a knowledge base, keyword matching to route requests, and retrieval to fetch context. Rebuilding it into what we run now is where most of my learning happened, so let me go through the parts that changed how I think.

The loop that runs alone

The first thing you learn building an autonomous system is that "autonomous" is a lot of unglamorous engineering. The agent runs on a cycle. Each pass it:

• Looks for new work
• Decides what to do
• Does it
• Records the outcome

It has to know when to retry a failure and when to stop and hand off. It has to tell a transient problem, like a database connection blipping, from a real one, and treat them differently, because retrying forever is as bad as giving up too soon. It even has to know when to be quiet, so it is not doing sensitive work at three in the morning unwatched. The loop is where autonomy actually lives. The model is a single step inside it.

The evals that tell you the truth

The second thing you learn is that you cannot improve what you cannot measure honestly. So a huge part of harness engineering is evaluation. We keep a frozen set of cases the system must handle correctly, and especially a set it must never act on. Every change gets scored against them before it ships. There is a precision wall for the write path that has to be perfect, or the release is blocked. And there is a regression guard that will not let a change go live if it breaks a case the current version passes, so teaching the system one thing can never silently make it worse at another.

What surprised me most is that these evals are not assertions we write by hand. They run the real code against real inputs and check what it actually decides. If someone weakens a safety check, cases quietly flip and the gate turns red on its own. The test cannot rot into agreeing with the bug. That idea, that your evaluation should be adversarial toward your own code, is the single most useful thing I have picked up.

The loop that improves itself

Then it gets genuinely fun. On top of the harness sits a research loop whose whole job is to make the system better on its own. It:

• Watches how the system is performing
• Notices when something slips or when a kind of request keeps getting missed
• Forms a hypothesis
• Proposes a change to the matching configuration
• Tests that change against the frozen evals
• Only promotes it if it clears every gate

If a promoted change misbehaves in the wild, it rolls back automatically. The part I find beautiful is the discipline around it. The loop is allowed to change how the system reasons about a request. It is structurally forbidden from changing what the system is allowed to do. All the dangerous surfaces - the write actions, the permission logic, the safety corpora - live in code and configuration the loop cannot touch. The type system is the first wall. So the system can get smarter without ever getting more dangerous.

Compile, do not retrieve

The last thing I want to mention is a lesson that flipped an assumption I did not know I had. We used to retrieve knowledge from a knowledge base at request time. Now we compile it and hand the model the whole thing at once. I assumed retrieval was obviously the right call - it is what everyone reaches for. But retrieval can hide the correct answer, and it did: as we taught the old system more, the right choice kept getting crowded out. Handing the model the full picture, and paying the extra tokens, fixed it. Correct beats clever, and clever is often just a nicer word for cutting a corner.

What this taught me

If you are getting into this, my honest advice is to fall in love with the boring parts. The loop, the retries, the evals, the deterministic scaffolding. That is where autonomy is won or lost. The model will keep improving on its own. The harness is the part you build, and it is the part that decides whether your system is a clever demo or something you can actually trust.

We built all of this together as a team, on open source foundations. I am going to keep writing up what I learn as I go. If you are learning this too, or building it, I would love to hear how you are thinking about it.