DEV Community

Your AI Agent Needs a Maintenance Window Protocol

Long-running agents are usually tested at startup and during normal operation. The awkward middle is ignored: what happens when you need to deploy a new image, rotate a credential, migrate a database, or restart the host while the agent is halfway through a tool call?

A process supervisor can restart a crashed agent. It cannot decide whether a browser checkout was committed, whether a webhook was acknowledged, or whether a tool call is safe to replay. That decision belongs in the agent runtime. This post presents a small maintenance-window protocol for agents that run for hours or days. It has four goals:

  • stop accepting new work;
  • let safe work finish or reach a checkpoint;
  • make ambiguous work visible instead of guessing;
  • resume with an explicit recovery decision.

1. Model maintenance as a state transition

Do not treat maintenance as kill -TERM followed by hope. Give the runtime a durable state machine:

RUNNING -> DRAINING -> QUIESCED -> STOPPED
                      |
                      +-> NEEDS_REVIEW
  • DRAINING rejects new jobs but allows an active job to continue until its next checkpoint or deadline.
  • QUIESCED means there are no unclassified side effects in flight.
  • NEEDS_REVIEW is the safe outcome when the process died after sending a request but before recording the response.

Persist the transition, not just an in-memory flag. A minimal record can look like this:

{
  "runtime": "agent-7",
  "maintenance_id": "mw-2026-08-10-001",
  "state": "DRAINING",
  "started_at": "2026-08-10T08:00:00Z",
  "accepting_work": false,
  "active_runs": 2
}

If the host disappears, the replacement process can see that the previous shutdown never reached QUIESCED. That is much more useful than inferring health from a missing PID.

2. Put checkpoints around side effects

An LLM step is usually replayable. A payment, email, browser click, deployment, or Git push may not be. Record a checkpoint immediately before and after every non-idempotent boundary:

PLANNED -> DISPATCHED -> ACKNOWLEDGED -> OBSERVED

On restart:

  • PLANNED can be dispatched again.
  • DISPATCHED without an acknowledgement becomes NEEDS_REVIEW unless the provider supports an idempotency key and status lookup.
  • ACKNOWLEDGED can be reconciled by reading the provider state.
  • OBSERVED is complete only when the runtime has stored the result it will use for the next decision.

An idempotency key should be derived from the logical operation, not the process attempt. For example, use invoice:8472:send, not a random UUID generated after every restart.

3. Make the drain deadline explicit

Every maintenance request needs a deadline and a policy for work that misses it. For example:

maintenance:
  drain_timeout: 90s
  on_deadline: checkpoint_and_stop
  unknown_side_effects: quarantine
  accept_new_work: false

For browser agents, checkpoint the URL, authenticated identity, page state hash, last submitted action, and external request ID. Do not claim that a page reload proves a form submission did not happen. Put the run in quarantine and reconcile against the application’s actual state.

4. Test the protocol with failure injection

A maintenance protocol is only real if you can interrupt it at each boundary. Run this small matrix in a staging environment:

Injection point Expected result
During queue drain No new job is accepted
Before tool dispatch Job remains replayable
After dispatch, before acknowledgement Run is quarantined or reconciled by key
After acknowledgement, before local write Provider lookup restores the result

| During

Read on DEV Community ↗ ← Back to News

Comments

No comments yet. Start the discussion.