Sentinel Dev Diary: Reading After the Build
Build step six of Sentinel is the rules engine: triggers, evaluation, warmup, input loss, loop protection, and the table of fixtures every rule has to pass. The document that owns it, The Rules Engine, was published here on 4 November.
The last entry was about reading before building. Step five's seam review ran the day before its first commit, so every gap it found became a change to the document, decided with no code depending on either answer. Step six went the other way. Its audit came after the code, and its seam review came nearly three weeks after the step was marked done. By then the code had already answered nearly half of what the review asked, because it had needed an answer and picked one. Several of the other questions turned out to be defects in code that was already marked done.
The step opened on 10 September, its audit was written and the step marked done on the 13th, and the seam review was written on 2 October. First, though, the finding the last entry held back. It is about a document that was published last Wednesday, and it was found before step six had any code at all.
Seven steps and no rules engine
The Core Runtime writes startup as a numbered list. As published:
- Connect to Postgres, acquire the advisory lock, or exit
- Increment and read the core epoch (below)
- Load vocabularies and registry
- Load and restore the projection
- Bind the JetStream durable consumer
- Start the watchdog, derived points and verification timers
- Start the command plane and the API
The rules engine is not in it, and neither is the claim set, the durable table of who wants each writable point in which state.
The seam review for The Command Plane noticed on 7 September and filed it as OI-42, an open question against the specification. It matters because The Rules Engine ties protection to a moment. A fail-safe interlock "asserts its fail_value the instant the core starts, before any inputs are live".
The Core Runtime's own section on what fails while the core is down says what that moment is worth: "No interlock protection. A fail-safe interlock is a claim held by a rule inside the core. If the core is not running, the rule is not evaluating and the protection is not being applied."
So protection begins when the rules engine starts, and the list did not say when that is. Read strictly, "the instant the core starts" is step one, before the registry that declares the rule has even loaded.
The Command Plane had the same gap one layer down. Claim expiries are recomputed "when the claim set loads", and nothing said when that happens either.
Step four as written restored only the projection, the core's in-memory record of the latest observed state of every point. What each device should be doing comes from the claims: arbitration resolves them into one desired value per point, and that lives in a different table with a different writer.
The order cannot be chosen freely. An interlock claim asserted before arbitration runs has nowhere to go. Arbitration that runs before the claim set loads resolves against an empty set and marks the point as wanted by nobody (no_intent) while a live claim wants it. And The Command Plane promises that at startup every claim is undeliverable and nothing is sent, which only holds if the claims load after the projection knows which drivers are up.
In the review's words, "There is a correct order and three plausible wrong ones."
It was settled the next day without adding a step. Step four was widened to restore the intent, meaning the claims and the desired values resolved from them, as well as the projection. Step six starts the rules engine with the other evaluators, so a fail-safe interlock asserts there, and its claim sits undeliverable until step seven brings up delivery. Step seven lost the phrase "the command plane", which had invited the reading that all of it starts last. Claims, arbitration and suspension are up by step six. Only the part that reaches a device waits.
The canonical document now reads:
- Connect to Postgres, acquire the advisory lock, or exit
- Increment and read the core epoch (below)
- Load vocabularies and registry
- Load and restore the projection, and the intent
- Bind the JetStream durable consumer
- Start the evaluators: the watchdog, derived points, rules, and the verification and claim-expiry timers
- Start delivery - the reconciler - and the API
The code already enforced an order. run_startup refuses steps that arrive out of sequence, against a tuple of seven steps in app/runtime/startup.py. That tuple was a second copy of the list, and nothing compared the two.
So the same change added a test that reads the numbered list out of The Core Runtime itself and asserts it matches the tuple, step for step. Its docstring gives the reason: OI-42 was three mechanisms with no place in this list at all - the rules engine, the claim-set load and the first arbitration pass - and it went unnoticed through four build steps because the list looked complete and nothing compared it to anything.
The test matches each step on a keyword rather than on its wording, so the prose can be rewritten freely and a step that moves still fails it.
A restart and a standing claim
Three passages in the published copy of The Rules Engine were settled on 11 September, while step six was being built. Two of them, on whether replay gates the step and how many loop limits there are, make no difference to what follows. The third does.
Claims are rows in a table, so when a core restarts, the rules' claims from before are still there, and every rule starts in warmup, which "does not evaluate". The only sentence about what a warming rule holds is the first of the extra test cases: "assert it holds no claims rather than evaluating on absent data".
Read as an instruction for a rule that has claimed something, that withdraws every rule claim at every restart, so each device moves to the next band for the length of warmup and back again once the rule is warm. The engine as built already took that reading.
The amendment says the opposite, and the engine changed with it: a standing claim is left alone through warmup, and the case now asserts both halves: no new claims, and an old one left standing. That comes back further down, because the fixture table could not tell the difference.
One rule
The fixture table runs every rule in the registry, and at the audit the registry had one. It is called aircon-cooling. It switches relay channel 8, which runs a cooling unit, from a derived point that goes high at 25 °C and low at 24 °C. It has no warmup deadline, no hold_max and no interlock class, so three of the four extra cases could not be exercised by anything that shipped. They are asserted against rules built inside the test files instead.
There was no interlock rule in the registry either, and there still is not. One of the findings below is an interlock check that was never built, and part of why it went unnoticed is that the only interlocks were in test fixtures, on a test point that already declared everything the missing check would have asked for.
The warning nobody built
Step six's audit was written on 13 September, by reading the code against the document. The document gives the step two criteria. The fixture table passed. The other is load-time validation, and the published copy is firm about it: "Load-time validation is the other half, and it is a test rather than a hope: a fixture rule set containing each of the listed violations fails the build with the right message, including the two that are errors rather than warnings."
The list printed here has eight bullets. By the audit, amendments had made it nine, and eight were enforced. One had no code at all:
- No two rules at the same band claim the same point with different values (warning, not error - sometimes deliberate)
It had no check, no reason code and no log line. A registry containing exactly the configuration the document says to warn about loaded without a word.
A decision from step five depended on it. Every claim carries one of six bands, from a default at the bottom to an interlock at the top, and the higher band wins. When step five built arbitration, it had to say what happens when two claims tie on band and on timestamp, and the answer it recorded was to break the tie on the owner's name, so the result is at least repeatable. It declined to treat the configuration as an error: "The owner tiebreak makes the outcome repeatable so it can be tested and explained; the lint warning is what is supposed to stop it happening."
No OI-*, because the mechanism is not silent here - it says this should not arise. That reasoning held up. The warning it relied on did not exist, so two disagreeing rules got a deterministic, explainable and entirely unannounced winner.
Two guards, both looking the other way
The audit also explains why nothing caught it, when two things should have. The first was a test that iterates every rejection reason the loader has and asserts each one has a case that fires it. A bullet that was never given a reason is invisible to it. It checks that the reasons that exist are tested, not that the reasons the document lists exist.
The second was the component register, which said the loader "refuses every bullet Rules Engine lists, one named reason per bullet with a fixture asserting each fires" and marked it green. The audit: "Nobody counted bullets against reasons; the sentence reads as a check and is a claim." That is the shape step five's audit named, a note that argues from what the code should do rather than from what it does, and the audit counts this as the fourth instance in a week.
It would be natural to fix it the way OI-42 was fixed, with a test that reads the document's list. The audit declined, because a test that counted the specification's bullets would be "a second home for the list", which is what the registers exist to prevent. The startup test gets away with it because the code's tuple has to exist anyway, one entry per step, and the test only compares two copies that are already there. The loader's reasons map to the bullets loosely. Some reasons serve several bullets, and some bullets are enforced by the schema before the loader sees them, so there is nothing to pair.
The correction went into the register instead, and the missing bullet was found the way it could have been all along, by someone counting by hand.
A warning that is not a rejection
The fix landed the same day. It is a check that logs and returns nothing. In this loader a reason code is a rejection: every reason is collected into the list that fails the load. Giving the warning a reason of its own would have turned the document's warning into an error. The shape was copied from the one other load-time warning the specification asks for, which also warns inline and rejects nothing.
Two choices came with it. Interlocks are left out, because two interlocks disagreeing is already refused as an error, and one disagreement reported twice under two severities leaves a reader deciding which applies. And two rules claiming the same value are not warned about, following the bullet's "with different values", because they arbitrate to the same answer whichever wins. The record of the fix: "warning there would train a reader to ignore the warning that matters."
Step six was marked done later that day, and the step-five decision about tiebreaks became true as written.
The criterion's last clause was settled the same day. It says every listed violation fails the build, "including the two that are errors rather than warnings". The list marks one violation as an error, one as a warning and the rest not at all, so there is no "two" for the clause to mean, and the one warning is the violation that must not fail the build. The document now says every violation reports its own message and every one except the warning fails the build, and that the warning is asserted on its message. A load that succeeds silently and one that succeeds having said why look the same to a test that only checks the exit code.
An empty list that meant two things
The same audit filed another row, and building it proved the row's own diagnosis wrong. The first of the four extra cases needs a rule that never leaves warmup to leave a claim it held before the restart standing. That is the September amendment above. The row said the fixture format needed a key for a prior claim. It did not.
The engine never sees standing claims. What it returns for a warming rule is "leave what stands alone", which is different from "assert nothing", and the fixture harness accepted expect_claims: [] against either one. So if a warming rule had regressed to withdrawing every claim on every restart, which is exactly the reading the amendment ruled out, the table would have gone on passing.
The fix was a new key, expect_untouched, and silence in a row became a failure. That turned the one shipped warmup row and two of the harness's own rows red until they said what they meant.
The seam review, after the fact
A seam review reads one document against the others it touches and asks what each assumes of the other, and where neither says. Step six's was owed from the day the step was marked done and was written on 2 October. Its introduction sets the terms: "This step is built, and that decides the routing. Step five's review filed every finding as an open question against the document, because no code existed to have taken a position. Step six's code had already taken plenty."
It had recorded eleven readings of its own while being built, as decisions in the register, and the documents had been amended for only a few of them, so much of the review's job was writing down what the code had decided and asking each document to agree or disagree. Two readings had never been written down anywhere, and the review wrote them. It filed nine questions and two pieces of owed work, and an addendum the same day filed two more questions. Four of the nine could be closed by ratify
Comments
No comments yet. Start the discussion.