Thirty years of confident wrong answers

I have spent thirty years walking into rooms where a distributed system was misbehaving in production and somebody was certain they knew why. Often they were wrong. Not stupid. Wrong, and confident.

I went to school for aerospace engineering. The first lesson of the field is margins: you never trust the nominal number, you trust the measured one, and you design for the gap between them. The second lesson is failure analysis: when something breaks, the explanation everybody likes is the least useful thing in the room until the data agrees with the explanation.

Then I spent a career as the person a customer calls when the database, the cluster or the network does something the brochure said was impossible. The title was usually Field CTO. The job was always the same. Find out what is true, say so out loud, and do not let anyone defend their work instead of fixing the work.

I have run into a lot of wrong and confident. The one I always remember is where I met someone who has become one of my best friends, and the relationship started with wrong and confident.

The Monday Morning Slowdown

Our team had spent two and a half years deploying a database on the company's shiny new internal private cloud, part of a new transformation project, and one day we hit it. Our newly deployed production database would slow to a crawl every Monday morning.

We spent weeks reading database logs and system logs, analyzing what in the workloads was causing the slowdown. We instrumented every database VM to see if we could pinpoint the failure, and after weeks, still nothing. The customer was upset, because now we were on the critical path, delaying the full production rollout.

Then one Monday, as I was flying in to this customer, I remembered something I had read a long time ago, Law 11 of Akin's Laws of Spacecraft Design: "Sometimes, the fastest way to get to the end is to throw everything out and start over." So that is what I did.

Back to First Principles

I went back to first principles: can I reproduce the error in a clean test harness? I stood up my own database and load tested it Monday to Friday, watching it perform properly, trying to see if I could identify the slowdown. On Friday I hopped on my airplane and went home.

Monday morning, working remotely, I grabbed my cup of coffee and started the load tests again, and there it was. The slowness reproduced in a clean test environment, with all the instrumentation in the world on the VM and nothing showing the root cause of the problem.

So we kept testing all week long. Friday came, I stopped my test, and on Monday I flew back to the customer. Sitting at my desk at the customer, I started the load test and hit the same problem again. Okay. This wasn't the database. Something was happening here that I couldn't measure or control.

So I walked into the infrastructure team's work area, sat down with the vendor's lead infrastructure architect and told him something was wrong with his infrastructure. He looked at me and told me there was nothing wrong with his infrastructure. They had deployed thousands of VMs and no one had identified anything like this problem.

I argued back: none of those thousands of VMs was running a persistence-based product like ours, so the problem had to be in their storage. That is when we started looking at the whole stack, from the VM hypervisor, to the virtual storage drivers, to the tiered storage platform, to the storage network, literally every layer, and instrumenting each and every one of them.

The Storage Tiering Policy

After months of reading logs and analyzing dashboards, one Monday we were debating the storage layer, and we noticed the access latencies. On Monday they started super slow. By Tuesday performance had improved by an order of magnitude, and by Friday it had jumped two orders of magnitude from the first measurement. What could cause such a jump?

The storage architect said the only thing that could do that was the smart storage policy on the storage platform. After months, that was it. The storage tiering policy would relegate lightly used blocks to SATA storage. As blocks got hot, it would move them to Fibre Channel storage, and as usage grew further, to NVMe. Over the weekend, when no tests were running, it moved them back to SATA.

We fixed it by simply pinning every database VM to a Fibre Channel storage tier. A simple solution to a confounding problem. When in doubt, instrument, test, and if you have to, start over.

Rules for Merlin

Here is what thirty years of those rooms taught me, written as the rules I now give everyone who works on Merlin, people and models alike. These are verbatim from the project's rulebook:

• "Never spin theories. Pull the actual data (exact log lines, decoded diffs, metric values) and verify every assumption by reading the code before stating a diagnosis."
• "A hypothesis is not a conclusion, and presenting one as fact wastes the user's time."
• "If we broke it, own it and fix it."
• "Say "I'm not sure" when uncertain rather than confabulating."

Read the last one again. I wrote the rule for an AI model. The rule fits half the war rooms I have stood in.

A language model is the most fluent confident-wrong colleague I have ever worked with. The model never says "I'm not sure" unless you build a system rewarding the sentence. The explanation sounds exactly as convinced when the explanation is wrong as when the explanation is right.

So I treat Merlin the way I treated every production system I was ever responsible for. I read the logs before I believe the summary. In June 2026, before deciding which fix to build next, I had the last ten self-review runs read directly from our log store, one reader per run, every verdict backed by a quoted log line. Two of the ten had finished cleanly. The other eight had ended in forced or budget-limited finals. The assumed blocker was the final gate. The logs said the blocker was tasks not finishing. We fixed the thing the logs pointed at, not the thing we expected.

This is just one of the experiences that have allowed me to be successful in my career, and it is why I built Merlin. AI and models are amazing technology, the stuff of science fiction novels. But as an engineer and architect, I live in a deterministic world. We have to have evidence for a conclusion, and we can't be afraid to be told no: the data doesn't support your conclusion, try again.

I have gone back to Akin's Laws a lot, and with AI and models I anchor on Law 17: "The fact that an analysis appears in print has no relationship to the likelihood of its being correct."

Nothing about this is new. A confident wrong answer is the oldest failure in engineering. The difference now is the speed and the volume. A model produces confident answers faster than any war room ever did. The discipline has to live in the system, because no human reads fast enough to apply the discipline by hand.