Sentinel: The Core Runtime

Single Writer

Run two Sentinel cores against the same message stream and nothing fails.

The durable consumer they both bind to splits the observations between them, so each builds its picture of the estate from half the data, and both report healthy.

The timer wheel in Watchdog & Derived Points, the single arbitration resolver in The Command Plane and the interlocks in The Rules Engine all assume this never happens, and none of them says what guarantees it. This part does.

Most of the rest is what having only one core costs. While it is down the message stream has to hold everything, so the buffers are sized for a four-hour outage, and how long the backlog takes to drain afterwards depends on how busy the core normally is: an hour at 20% utilisation, sixteen hours at 80%.

When it falls behind there is no second core to share the load. A core that is down protects nothing, so protection that has to survive it belongs in a local device and not in a rule.

Nearly every figure here is a design target rather than a measurement, and the document says so. The one the rest of the arithmetic depends on, how many observations a second the core can sustain, is still unmeasured, and the exit criteria end by asking for it, so that the four-hour figure can be checked instead of assumed.

Scope

What this document owns:

• single writer as a requirement and what breaks when two cores run
• the advisory lock and the startup order that takes it first
• failing fast when the lock is lost
• the core epoch and fencing the bus
• the scale targets and the observation rates behind them
• the difference between a planned restart and a tolerable unplanned outage
• buffer sizing and the drain arithmetic
• how the system behaves during a drain, including the pause on warmup deadlines
• backpressure, meaning lag read from the consumer's pending count, a full stream that rejects rather than discards, and the poll schedule as the only valve
• what stops while the core is down
• restart time
• the rule against interlocks at a site that can be partitioned from the core
• multiple cores as a recorded non-goal

It does not own the staleness suppression mechanism and its constraints, the startup grace, the recovery rule for late observations, or the strategies for bounding rehydration, which belong to Watchdog & Derived Points; the driver's epoch tracking and its local disk queues, which belong to Driver Contract; what the core does with a stale_epoch rejection, which belongs to Command Plane; the input-loss policies and warmup themselves, which belong to Rules Engine; or retention and compression, which belong to Retention & Compaction.

The Core Is a Single Process

The core is a single process, and exactly one instance of it runs at a time. Single writer is a requirement, not an assumption.

Five mechanisms already assume it: the in-memory projection is authoritative, the timer wheel holds freshness deadlines, the JetStream consumer is durable and single, verification timers are core-side, and the arbitration resolver is the only writer of point_desired.

Two cores would not degrade gracefully. They would:

• Split the observation stream rather than duplicate it. Two consumers bound to the same durable consumer each receive a share of the messages. Each core would build a projection from half the data and believe it complete. This is the worst of the failure modes because nothing errors - both cores look healthy and both are wrong.
• Double-fire commands. Two arbitration resolvers reaching the same conclusion issue two writes; reaching different conclusions issue contradictory ones.
• Duplicate timer firing. Two watchdogs expiring the same deadline, two verification timers on one command.

Because the first failure is silent, the requirement is enforced mechanically. A convention that only one runs is not enforcement, and the day it is violated is the day someone starts a second instance to test something.

Enforcement: An Advisory Lock, Not a Lease

A Postgres session-level advisory lock, held on a dedicated connection for the process lifetime, taken before anything else happens.

This is preferred over a lease in NATS KV for one reason: there is no timeout to tune. A lease requires choosing a renewal interval and an expiry, and getting that wrong produces either a slow failover or a window where two cores both believe they hold it. An advisory lock is released by the database when the connection dies, so liveness is delegated to TCP and to Postgres rather than to a chosen number.

Acquisition order at startup:

1. Connect to Postgres, acquire the advisory lock, or exit
2. Increment and read the core epoch (below)
3. Load vocabularies and registry
4. Load and restore the projection
5. Bind the JetStream durable consumer
6. Start the watchdog, derived points and verification timers
7. Start the command plane and the API

Nothing may bind the consumer before the lock is held. Binding the consumer (step 5) only after taking the lock (step 1) is the whole reason for writing this list down. A core that binds first and locks second has already stolen messages from the running instance before it discovers it should not be there.

A core that cannot acquire the lock exits non-zero immediately. It does not wait, retry in a loop, or start in a read-only mode. Under a supervisor this produces a restart loop, which is visible; a silently idle second core is not.

Losing the Lock While Running

Fail fast. On detecting that the lock is gone - connection death, or a periodic ownership check - the core stops publishing commands, stops consuming, and exits non-zero. No draining, no finishing the batch in flight, no degraded read-only mode.

A core that has lost the lock has, by definition, possibly been replaced. Anything it does from that moment is a second writer.

An uncommitted batch is not a loss: the messages are unacked and will be redelivered, and the dedup key makes redelivery harmless.

Fencing the Bus

The advisory lock protects Postgres. It does not protect NATS, and the dangerous window is exactly the one it cannot cover: core A is partitioned from Postgres, its lock is released, core B acquires it and starts, and core A has not yet noticed because its TCP socket has not timed out. For that interval, core A can still publish on cmd.>.

So commands carry a core epoch: a monotonically increasing integer stored in Postgres and incremented on each successful lock acquisition. Drivers track the highest epoch they have seen and reject any command carrying a lower one, with outcome rejected and reason stale_epoch.

This is one integer in the command envelope and one comparison in the driver, and it is the only thing standing between a partitioned core and a relay pulsing on the instructions of a process that has already been replaced.

The epoch is also stamped on every control_event, which makes "which core did this" answerable after the fact. Drivers do not need to know what an epoch means, only that it never goes backwards.

Scale Targets

Every number below is a design target rather than a measurement. They exist so that the sizing arithmetic has terms, and so that a future measurement has something to disagree with.

• quantity target devices: 200+
• network segments, and therefore driver hosts: 16+
• device timeseries points: 2,000+
• derived points: 1,000+
• self-points: ~700
• total points: ~4,000

Self-points are the points the system publishes about itself. Their figure is the one most easily forgotten, and it is not small: roughly two per device, seven per driver host, one convergence point per writable point, and the core's own signals. They are points in every respect, so they consume registry entries, timer-wheel slots, storage and ingest capacity like any other.

Observation rate. Three tiers, and the middle one dominates:

• a handful of points at 1 Hz - liveness, air quality, heart rate
• the bulk at roughly 10 s
• a few hundred hourly

That works out at ~220 observations/second steady state, of which the 1 Hz points are around 5%. The design figure is 500/s, leaving room for the fleet growing and for the 10 s tier tightening, which is the parameter the whole figure is sensitive to: at 5 s the rate doubles.

Storage figures, all following from the above:

• ~19M raw rows/day, ~40 GB rolling under a 90-day retention with compression from day seven
• ~5.3M rows/day in the minutely continuous aggregate, which is the long-term storage driver rather than the raw table
• ~4,000 timer-wheel entries, well inside a single wheel's design
• ~1,000 rehydration queries at cold start, which is the restart time almost in its entirety

Sustainable ingest throughput is the one term still unmeasured, and it is the term utilisation depends on. Per-observation validation, registry lookup and unit conversion in application code is where the ceiling sits, not in Postgres. It is measured against the 500/s design figure before the stream is sized, rather than assuming the batch insert path is the limit.

Two Different Outage Numbers

These are not the same quantity, and conflating them sizes the buffer against the wrong case.

Planned restart is lock release plus cold start: minutes, dominated by derived-point rehydration. This is what an upgrade costs, and Restart time below covers it.

Tolerable unplanned outage is how long the core can be down without losing data - hardware failure with nobody around, or a fault in the middle of the night. The working figure is four hours, with the expectation that the real case is usually far shorter. Only the second sizes the buffer.

Sizing the stream against restart time would leave the system losing data in exactly the scenario it cannot control.

Buffer Sizing

While the core is down, drivers publish normally and the observation stream absorbs everything. The stream is the first buffer; the drivers' local disk queues are the second, engaged only once the stream is full and rejecting.

The stream is bounded by bytes and messages only, never by age. A max_age limit deletes the oldest messages silently once they pass the age, regardless of whether anything has consumed them, which is precisely the middle-of-the-chain loss that rejecting on full was chosen to avoid (see Where the loss goes below).

The stream size is derived from observation rate, not expressed as a duration limit.

Recovery is set by headroom, not by outage length. After an outage of length O, the core drains the backlog while new observations keep arriving. With C the sustainable core throughput, and steady-state utilisation u the ingest rate divided by C, the drain takes O × u/(1-u) and the peak backlog reaches O/(1-u) of steady-state volume.

For a four-hour outage:

• utilisation drain time buffer needed 20% 1 hour 5 hours 50% 4 hours 8 hours 80% 16 hours 20 hours

At 80% the system takes four times the outage to recover and needs five times the outage in buffer, which is how a system that looked fine in steady state turns a short fault into most of a day. Recovery within an hour of a four-hour outage requires running at roughly 20%.

At the scale targets above, this is comfortable, and the stream is sized generously rather than carefully. Provided sustainable throughput is anywhere above about 1,000/s, utilisation sits under 25%, a four-hour outage drains inside 80 minutes, and the buffer requirement is roughly 5.3 hours of steady state - 8M messages and 4 GB, which is cheap enough that rounding up costs nothing and buys margin against the throughput figure being worse than assumed.

Two refinements:

• Drain throughput is usually better than steady-state throughput, because batches fill completely rather than flushing on the 100 ms timer. So treating C as the steady-state figure is conservative in the right direction.
• The total buffer is the stream plus the sum of the driver queues, so the requirement can be split between them. The stream is the better place for it, though, because a driver queue filling means that host is also at risk of losing data on its own bound.

Draining

The system behaves correctly during a drain, and this is worth stating because it looks alarming. Replayed observations carry their original observed_at, which is older than the current deadline, so under the recovery rule in Watchdog & Derived Points they do not clear staleness. Points do not reach live until the backlog is worked through and genuinely current data arrives.

Rules stay in warmup and the system declines to automate until it has caught up, rather than making four hours of retrospective decisions in ninety seconds. That is the design working rather than a fault.

Two things must hold for it to keep working:

• Staleness evaluation is suppressed while lag is high, as described below, so the drain does not also produce a fleet-wide staleness cascade.
• Warmup deadlines pause under the same condition. warmup: deadline(60s) exists to stop a rule waiting forever for an input that is never coming. During a drain the input is coming, just not yet, and letting the deadline expire makes rules evaluate under their input-loss policy while the data they need sits in the queue. Safe, but premature, and for the wrong reason. The pause is the same principle as the suppression: a core-internal condition must not drive device-level decisions.

Backpressure

Single writer determines the shape of this before any policy is chosen: the usual answer is unavailable. When a consumer cannot keep up, the standard response is to add consumers, and there is exactly one core by construction. Backpressure here is solved by shedding load or by going faster, never by scaling out.

Lag is already measured, for free. Because ingest acks after commit, the durable consumer's pending count is the lag, and the age of its oldest unacked message is the second half of the picture. No separate instrumentation is needed and no watermark table exists to disagree with.

A backlog survives a restart. Resuming from the ack position is what makes restart correct, and it also means a backlog cannot be cleared by bouncing the core.

Draining or discarding is a deliberate operation, never a