Delegating to AI Means Governing the Environment

In the previous article, I argued that AI isn't simply changing the tools we use to develop software, but shifting our work to a new level of abstraction. In this one, I want to address the problem that immediately follows: if we're going to write less and less code directly and agents are going to produce an increasingly larger part of it, how the hell do we know whether what they code is actually right?

Because the answer obviously can't be "trust the AI, it's very smart". Even though I personally develop code with AI today with practically no review, I don't blindly trust AI. Just as I don't blindly trust an engineer on my team. I don't even blindly trust myself. Blind trust is a security hole.

And not blindly trusting someone doesn't mean distrusting them, it means having mechanisms to prevent their mistakes, or mine, from causing problems. That's why we've spent decades building mechanisms and methodologies around software development to detect, and avoid as much as possible, our mistakes:

• XP
• Scrum
• Tests
• Code reviews
• Pair Programming
• CI
• Static analysis
• Permissions
• Observability
• Environments
• Containers
• Auditing...

The question, therefore, shouldn't be whether we can trust an AI. The question should be what system do we need to build so we can use it without needing to blindly trust it?

It's not deterministic

One of the first objections is usually that if you ask it the same thing twice, it generates two different pieces of code. True. But if you give the same task to two different programmers, or to the same programmer with enough time in between, we'll very probably get two different implementations too, depending on the complexity of what we're asking. And if we've never required two developers to produce exactly the same code, why do we expect AI to produce exactly the same code from the same request?

Isn't it enough for the result to satisfy the requested requirements? That it does what it's supposed to do. That it passes all kinds of tests. That it satisfies the acceptance criteria. That it meets security requirements. That it respects certain constraints.

The problem appears when we keep evaluating the writing process while this new level of abstraction increasingly forces us to evaluate the result against a contract. And when I say contract, I'm not talking about a huge spec telling the agent how it has to build every single thing. I'm talking about the conditions the result must satisfy and having mechanisms that can verify them.

We haven't reached compiler-level reliability yet, and I don't know whether we ever will, or even whether we need to. What is happening is that agents are becoming more capable and we're giving them more and more autonomy. And the more autonomous they become, the less sense it makes to think we'll be able to control them by constantly looking over their shoulder.

Rules can't live inside the agent

This is, for me, one of the important parts. A rule written only in a prompt isn't a rule. It's a request. You can have an AGENTS.md, a skill or whatever you want telling it never to do a certain thing. It helps enormously. I use them, obviously. But we've already seen that the agent forgets them more often than we'd like, sometimes interprets them incorrectly, can even lose that information among the ever-growing amount of context it's handling, or simply finds another way to achieve what it was trying to do.

And that last one isn't theoretical. While experimenting with all this, I configured a mini PC with Ubuntu to run a model's CLI. I removed the rm command from the system so it wouldn't be able to delete anything. Then one day I discovered that it had deleted files. How? I couldn't believe it. I asked, and it explained that since rm didn't work, it had used rsync --delete. I was surprised and scared at the same time.

I had prevented it from using rm. And even though I had explicitly told it never to delete a file without my approval, at one point it realized it had made a mistake and needed to rebuild a structure, so it decided it needed to delete things. It interpreted my prohibition as meaning it shouldn't delete anything critical and that, given the context it had, it could decide that deleting those files wasn't going to cause any problem. After all, it was code that hadn't existed before and that it had created itself, right? It disobeyed an instruction in order to achieve its goal.

That's why important restrictions have to live outside the agent, in mechanisms it doesn't control:

• Hooks
• CI
• Permissions
• Tests
• Mutation tests
• Policies
• Gates...

If I want every change to have tests, I can tell it to do TDD. But I also have to verify that those tests exist, that they run and that they actually pass. If I don't want any change to get in without review, I can't let the same agent decide that "this time" the review isn't necessary. If I want a security vulnerability to block a change, no second agent should be able to casually decide "well, it doesn't look that serious". If I want all work to be traceable, it can't depend on an agent remembering to create a ticket afterwards.

Important rules have to become mechanisms. Not good intentions. The agent can decide how to solve the problem. What it shouldn't be able to decide is whether it can bypass the mechanisms that determine whether that solution is acceptable.

Orchestrating isn't governing

And here I think we're usually mixing two different problems. One is how we orchestrate agents. Who writes. Who researches. Who reviews. How we share context. How several agents work in parallel. Which model we choose for each task.

We can have one writing, another from a different family reviewing, and a third arbitrating when they disagree. We can even launch the same task to several models and then choose between the solutions that satisfy our conditions. This already looks much more like building an artificial team than using a programming assistant.

But there's another different problem: who governs those agents? Who establishes the rules? Who decides which actions are allowed? Who checks that they were actually followed? Who prevents an agent from granting itself an exception?

Orchestration gives capability. Governance gives guarantees. And we need both.

What about specs?

For a while it seemed that the answer to AI development was the prompt engineer. It lasted months. Then the idea emerged that development with agents was going to be about writing better specs. Right? If the AI made mistakes, it was because we hadn't specified clearly enough what we wanted. So we started writing increasingly detailed documents explaining which endpoints to create, which classes to use, which structure to follow, which cases to handle...