Faster PRs, Weaker Instincts: The Judgment Problem in AI-Assisted Engineering

I thought the dashboard was telling me good news. My team had adopted AI-assisted coding quickly, and for a while it looked like exactly what everyone promises. More tickets closed. Shorter cycle times. More PRs moving. Same team, more output, less drag. I felt good about it. That was the part I got wrong. The moment it changed for me The moment this changed for me was not some giant outage or dramatic failure. It was quieter than that, which honestly made it worse. We shipped a change that passed every test, cleared review, and deployed cleanly. Two weeks later, we found it had been degrading a retry path under load conditions our test environments did not replicate. When I asked the engineer to walk me through the change, they could. The code made sense. They understood what each part did. But when I asked what would happen if the downstream service was slow, not down, just slow, there was a pause. Not because they were weak. Because they had never needed to ask that question. The AI wrote the handling code, the tests passed, and the whole thing moved forward before that question ever came up. That was the moment the dashboard stopped telling me the truth. Not because it was lying. Things really were moving faster. It just was not showing me the part that mattered. What I didn't see coming What I did not see coming is that AI can make a team look more capable before it makes the team actually more capable. I used to learn by getting stuck. Not the fun kind. The 11 PM kind. The kind where you stare at a stack trace for hours, chase the wrong theory three times, and slowly build the instinct that tells you something technically works but still is not safe. That instinct did not come from polished answers. It came from friction. From getting burned. From seeing what fails under pressure. That is the part I do not think we have really replaced yet. When the first draft is free, a lot of early learning gets skipped. You still get output. You still get motion. But you do not automatically get the judgment that used to be built inside the struggle. Maybe that gets rebuilt somewhere else. Maybe I am missing it. But I have not seen it happen on its own. The gaps don't show up all at once The first gap usually shows up in review. One engineer can now generate the kind of PR volume that used to take several people. Reviewers get overloaded. Everyone starts reading faster. If the tests pass and nothing looks obviously broken, it probably gets approved. I started seeing PRs where the tests were present, clean, and completely disconnected from the actual failure mode. That is a very specific kind of problem. It looks like rigor. It is not rigor. The next gap shows up in debugging. If every error can be pasted into a chat window and turned into a plausible fix in thirty seconds, you stop asking why it broke. The symptom gets patched. The lesson does not land. Then the next similar failure shows up in a slightly different shape, and the team takes longer than it should because nobody really learned the first one. And then the first real incident hits, and that is where you find out what did or did not get built. I had an engineer who had shipped well for months. Solid PRs. Fast delivery. Good feedback. Their first real incident was rough. Not panic. They just had not done enough time inside a broken system to know how to narrow things down when the answer was not obvious. That takes reps. I do not know another way to say it. What we actually tried We did not solve this cleanly. We tried one thing that did not work at all: adding more structure to standups and asking people to explain what they shipped. Mostly that just taught people how to sound confident about things they did not fully understand. What helped more was changing what code review was for. We stopped treating review as a gate and started using it as a place where people had to show their thinking. Not just what the code does. What they assumed. What fails first. What they chose not to handle. For high-impact areas, we started expecting one short paragraph in the PR description: what can go wrong here? That felt slower at first. It really was not. The questions people asked before opening the PR got better. The reviews got better. The code got better. We also started making room for practice that normal sprint work does not create on its own. Debugging drills. Postmortem walkthroughs. Getting junior engineers into incident shadowing earlier than felt comfortable. None of that is flashy. None of it looks like AI transformation in a slide deck. But it did make the speed feel more trustworthy. What I still haven't figured out I am still not sure how to measure any of this well. I do not have a dashboard for judgment development, and I do not fully trust anyone who says they do. I also do not think the answer is to back away from AI. That is too simple, and it misses what is actually happening. The tools are useful. They help people attempt things they would not have attempted before. They reduce some of the shame around not knowing. That part is real too. But there is a kind of understanding that only shows up when the system is breaking, the logs are ambiguous, and nobody has a clean answer yet. That is still learned the hard way. And I think a lot of teams are going to rediscover that at the exact moment they are congratulating themselves for moving faster. Open for discussion I am genuinely curious how other teams are seeing this. Are you seeing the same gap between output and understanding? Where does it show up first for you: review, debugging, incidents, or somewhere else? Have you found something that actually helps build judgment in an AI-assisted team? And the harder question: are we measuring what matters, or just what is easy to count? Top comments (0)