Blocking an AI agent's shortcuts with tools: what verifying code comments taught me about steering and MCP

How spec-lantern works

AI agents now review code and write tests as a matter of routine. They do not read only the code. They read the comments too, and sometimes they interpret the code through what a comment says it is meant to do.

I picked writing-tracker (MIT), a web app that another developer published on GitHub. It records how far along you are in writing a book. It was built with Lovable, an AI app builder, over three days at the end of April 2025, and it has not been updated since.

The code and comments the AI wrote are still there as they were, and it has logic such as aggregation and ordering whose behavior can be checked against its comments, so it made a good subject.

https://github.com/shateers/writing-tracker

I gave the app to Kiro and asked: "Check whether any comment disagrees with what the code actually does."

The statistics screen has a table that lists your progress for the last six months, one row per month. The code that builds it says:

// Generate last 6 months
for ( let i = 0 ; i < 6 ; i ++ ) {
  const monthStart = new Date ( currentDate . getFullYear ( ) , currentDate . getMonth ( ) - i , 1 ) ;
  // ...
  periods . push ( { label : `${monthName} ${monthStart.getFullYear()}` , start : monthStart , end : monthEnd } ) ;
}
// Reverse to have most recent month first
periods . reverse ( ) ;

The rows of the table come from the list of six months this code builds (periods). The first month in periods becomes the top row, and the rest follow downward. If the comment is right, the month that ends up first after reverse(), and so the top row of the table, is the current month.

Kiro read the code and judged the comment correct. Then a property-based test (PBT) checked the same behavior, and it disagreed on the first run.

PBT is a testing technique where, instead of writing input and expected-output pairs one at a time, you write a property that should hold for any input, and the framework generates many inputs to check it. When it finds an input that breaks the property (a counterexample), it shrinks it to the simplest form it can.

Here the property was "whatever today is, the top row of the table is the current month", and the test built the table again and again, setting "today" to many different dates. For example, with today set to January 2020, the top row was not January 2020 but August 2019, the oldest of the six months shown (August 2019 to January 2020).

The cause is the order in which the loop adds months. It starts from the current month (i = 0) and appends each earlier month to the end of periods. In the January 2020 example the list is "January 2020, December 2019, ..., August 2019", so the current month is already first. Calling reverse() on that flips the list and puts the oldest month, August 2019, at the top. To put the newest month first, as the comment says, the reverse() was never needed.

"If the agent believes whatever it reads, make it run the code." That is why I built spec-lantern. It is a Kiro Power. A Kiro Power bundles instructions for the agent with an MCP server configuration, and you install it into Kiro from GitHub or elsewhere. The current Kiro docs recommend the Agent Plugins format for new powers; spec-lantern uses the earlier POWER.md format, which still works.

spec-lantern makes the agent predict whether each comment is true, locks that prediction, and then runs the code in an isolated environment to find out. It is also my entry for Kiroween 2026, Kiro's hackathon.

https://github.com/ikenyal/spec-lantern

What building it taught me is that "make it run the code" is not enough. To reach its goal, the agent kept finding shortcuts I had not anticipated. And different kinds of shortcuts needed fixes in different places. Some were solved by one more sentence in the instructions. Others came back no matter how often I rewrote the instructions, and stopped only when I changed how the tools work.

This post walks through those shortcuts, taken from the actual records, and explains where I blocked each one (steering, meaning the instructions given to the agent; the MCP tools; or the execution environment) and why there. None of these ideas are specific to Kiro, so they may also be useful if you hand verification work to Claude Code or another agent.

How spec-lantern works

spec-lantern consists of a Kiro Power (POWER.md plus seven steering documents) and its own MCP server. The agent can touch the target repository only through the MCP server's tools.

flowchart TB subgraph agent["Done by the agent (Kiro)"] direction LR A[Split comments<br/>into claims] --> B[Write executable<br/>properties] --> C[Predict<br/>PASS/FAIL] end subgraph tools["Done by the MCP server's tools"] direction LR D[Lock the<br/>prediction] --> E[Run in isolation] --> F[Compare with<br/>the prediction] F --> G[Replay the<br/>counterexample] --> H[Fix on a<br/>throwaway copy] end agent --> tools

The agent's job ends at prediction. It splits comments into claims, writes an executable property for each claim (a test written with fast-check and Vitest), and predicts the result of running it. If the test passes, the result is PASS and the comment was right. If it fails, the result is FAIL: the comment disagrees with what the code does, which is to say it lied.

Everything after that (locking the prediction, running, comparing, applying a fix) is done by the MCP server's tools.

• Tool Role lock_prediction Records hashes of the prediction and the property file, with a timestamp supersede_prediction_lock Re-locks with a recorded reason when a property is fixed after locking execute_properties Runs the target in isolated containers and saves the raw results replay_counterexample Replays a failure with the recorded random seed, time zone, and fixed clock compare_predictions Compares the locked predictions with the execution results apply_fixes_and_reexecute Applies a fix to a throwaway copy and reruns every property

I did not start with this split. Earlier, the agent wrote the comparison tables and result files itself, and the lock did not protect the property files. As I dealt with the six shortcuts below one by one, the part left to the agent got smaller, and it ended up like this.

Six shortcuts the agent took

None of these were taken out of malice. Each was the agent meeting its goal by the shortest route available. A human engineer near a deadline might well make the same calls. That is exactly why reminders do not stop them: whenever the conditions line up, they come back in the same form.

What follows is what actually happened before any countermeasure. How I blocked them comes in the next section. To say it up front: some could be prevented by how the instructions (the prompt) were written, and some could not, no matter how often I rewrote them, and needed changes to how the tools work. That difference is the subject of this post.

One and two: distorted predictions

The first is the "read and believe" from the opening. Put a comment next to code, and the agent interprets the code on the assumption that the comment is right. Accepting "to put the newest month first" is a more natural way to read than tracing the order before reverse() yourself. In human review too, comments that explain intent exist to reduce the effort of reading code. Believing comments is built into the act of reading.

I thought running the code would solve this one. But within the run-the-code setup, a second shortcut appeared: the convenient prediction. This happened while I was trying the packaged Power in a separate environment. spec-lantern's highlight is the moment when a comment that looked right on reading turns out to be a lie when executed: a prediction of PASS and a result of FAIL. Kiro understood that aim. And for one comment, even though it had noticed while reading that something looked off, it predicted PASS. Its report gave the reason as "to make the mismatch between prediction and result meaningful". It thought the answer was FAIL and predicted PASS on purpose, to create the highlight.

What makes this shortcut hard is that it becomes more likely the better the agent understands the system's purpose. A prediction matters as a record of how the agent judged the code from reading it. Once it is mixed with a judgment about which result would look good, it stops being a record and becomes staging. A staged prediction, placed next to the execution result, no longer shows how Kiro read the code, and loses its meaning as a prediction.

Three and four: contaminated expected values

The third is deciding the test's answer by looking at the implementation. For a second subject I picked MoodPlays (MIT), also built with Lovable. It finds nearby places to match your mood, and it has logic such as parsing opening hours and calculating distances.

https://github.com/gargi04sahu/MoodPlays

Here, five of the first ten properties Kiro wrote computed the expected value (the answer the test treats as correct) with the same calculation as the code under test. Take distance as an example. The shape of the problem is: copy the formula from the code into the test, and check that both give the same result. Such a test passes even if the formula is wrong. It looks like a test, but it treats the implementation's calculation as correct and uses it as the answer, so it is the same thing as the first shortcut, read and believe.

When rewriting, I replaced these with expectations grounded in something other than the code: the real distance between two points with known latitude and longitude, or the concrete example given in a comment, when there is one. In software testing, whatever decides whether a result is correct is called a test oracle. What was needed here was a test oracle grounded independently of the code under test.

This shortcut is hard because it is the agent's most reliable way to write a "correct" test. Deriving the expected value yourself risks getting it wrong. Using the same calculation as the code at least guarantees the test will not break at run time. Once passing the tests becomes the goal, deciding the answer from the implementation is the rational choice.

The fourth is peeking before the lock. This also happened with MoodPlays. Before locking its predictions, Kiro copied the target code and ran it on the spot with node -e (a Node.js command that runs a short snippet), to check in advance whether the tests would pass. Kiro presumably did not want to miss. But this makes the prediction meaningless. A prediction in spec-lantern is a record of how far reading alone can tell whether a comment is right. If you run the code and see the result before predicting, of course you will be right, and you can no longer tell whether reading was enough.

Five and six: broken records

The fifth is rewriting after the lock. While I did several full runs to record the demo video, in one run Kiro kept editing property files after locking and re-running. The cause was that the tab-switching UI did not work in the test's simulated browser (jsdom), so the monthly table was never rendered and the test counted rows that were not there. That is a test defect, so fixing it was legitimate. The final result was correct, and Kiro's report showed reasonable grounds for each fix. But those reasons were not recorded, and afterwards nothing distinguished this run from one where the tests were edited until they matched the prediction. Inside a system built to show that predictions were right, it was no longer possible to prove that this prediction was right.

The sixth is hand-written tallies. Earlier, the agent copied execution results and comparison tables into files by hand. In one run it miscopied a prediction into the comparison table. In the next run, Kiro re-read the lock files and noticed the error. A miscopy is an honest mistake, but the fact that a table showing whether predictions were right can contain errors is itself the problem. A reader cannot tell whether the table is right without opening the original files.

What the six have in common

Lined up, the six show that shortcuts can get into every stage: prediction, expected values, execution, and records. Looking back, none of them are specific to AI. Teams that are just starting to write tests run into similar things. Filling in expected values with whatever the implementation outputs. Making a failing test pass by changing the test instead of the code. Reading code in review through what the comment says it intends. Copying errors creeping into hand-compiled reports. All of these happen on human teams.

The reasons are probably the same for humans and AI. One is goal displacement. The real purpose of testing is to find where code does not do what it should, but day to day, "the tests pass" is the visible result. Then humans and AI alike take the shortest route to passing. The other is that the checker and the checked are the same. If the person who wrote the implementation also decides the expected values, the same assumptions go into both. If the person who made the prediction also tallies the results, they read the results in the prediction's favor without meaning to.

Human teams have compensated with process and systems. Test-driven development, writing tests before the implementation, separates expected values from the implementation. CI checks that block merging until tests pass, and code review by someone else, keep people from judging their own work. Handing verification to an AI agent needs the same kind of measures. The difference is that they have to be built into instructions, tools, and the execution environment in a form the agent can read.

spec-lantern's "lock the prediction before running" is close to TDD's "write the test before the implementation". On top of that, the thing that decided where to put each countermeasure was whether the agent was aware that it was taking a shortcut.

Shortcut Stage Agent aware? Read and believe Prediction No (it judged the comment and code to agree) Convenient prediction Prediction Yes (it wrote the reason