Your AI Agent Needs a Maintenance Window Protocol
Long-running agents are usually tested at startup and during normal operation. The awkward middle is ignored: what happens when you need to deploy a new image, rotate a credential, migrate a database, or restart the host while the agent is halfway through a tool call?
A process supervisor can restart a crashed agent. It cannot decide whether a browser checkout was committed, whether a webhook was acknowledged, or whether a tool call is safe to replay. That decision belongs in the agent runtime. This post presents a small maintenance-window protocol for agents that run for hours or days. It has four goals:
- stop accepting new work;
- let safe work finish or reach a checkpoint;
- make ambiguous work visible instead of guessing;
- resume with an explicit recovery decision.
1. Model maintenance as a state transition
Do not treat maintenance as kill -TERM followed by hope. Give the runtime a durable state machine:
RUNNING -> DRAINING -> QUIESCED -> STOPPED
|
+-> NEEDS_REVIEW
DRAININGrejects new jobs but allows an active job to continue until its next checkpoint or deadline.QUIESCEDmeans there are no unclassified side effects in flight.NEEDS_REVIEWis the safe outcome when the process died after sending a request but before recording the response.
Persist the transition, not just an in-memory flag. A minimal record can look like this:
{
"runtime": "agent-7",
"maintenance_id": "mw-2026-08-10-001",
"state": "DRAINING",
"started_at": "2026-08-10T08:00:00Z",
"accepting_work": false,
"active_runs": 2
}
If the host disappears, the replacement process can see that the previous shutdown never reached QUIESCED. That is much more useful than inferring health from a missing PID.
2. Put checkpoints around side effects
An LLM step is usually replayable. A payment, email, browser click, deployment, or Git push may not be. Record a checkpoint immediately before and after every non-idempotent boundary:
PLANNED -> DISPATCHED -> ACKNOWLEDGED -> OBSERVED
On restart:
PLANNEDcan be dispatched again.DISPATCHEDwithout an acknowledgement becomesNEEDS_REVIEWunless the provider supports an idempotency key and status lookup.ACKNOWLEDGEDcan be reconciled by reading the provider state.OBSERVEDis complete only when the runtime has stored the result it will use for the next decision.
An idempotency key should be derived from the logical operation, not the process attempt. For example, use invoice:8472:send, not a random UUID generated after every restart.
3. Make the drain deadline explicit
Every maintenance request needs a deadline and a policy for work that misses it. For example:
maintenance:
drain_timeout: 90s
on_deadline: checkpoint_and_stop
unknown_side_effects: quarantine
accept_new_work: false
For browser agents, checkpoint the URL, authenticated identity, page state hash, last submitted action, and external request ID. Do not claim that a page reload proves a form submission did not happen. Put the run in quarantine and reconcile against the applicationβs actual state.
4. Test the protocol with failure injection
A maintenance protocol is only real if you can interrupt it at each boundary. Run this small matrix in a staging environment:
| Injection point | Expected result |
|---|---|
| During queue drain | No new job is accepted |
| Before tool dispatch | Job remains replayable |
| After dispatch, before acknowledgement | Run is quarantined or reconciled by key |
| After acknowledgement, before local write | Provider lookup restores the result |
| During
Comments
No comments yet. Start the discussion.