When "Not Found" Doesn't Mean "Never Happened"

Verified on 2026-11-09 against strands-agents 1.63.0, Python 3.11 and Bedrock in us-east-1. Every number below came out of a run, and the scripts live in usecases/05-reservations-fencing/src/ with raw output in evidence/. The model in the agent experiments is amazon.nova-pro-v1:0.

A reader of article 03 left three questions under the piece, and each one tightened on the last. A limit counts what happens inside one call, so what stops two concurrent attempts from each spending a full remaining quota? When a request was submitted but the response was lost, how do you know that "not found" means "never happened"? And when the crash lands before or after the side effect, is returning the capacity safe in both cases?

Those questions share one blind spot. Limits, leases and lookups all answer questions about a moment in time, which is why the answers stop being true the moment after they are given. This post builds the smallest system that fails, fails it on purpose and then makes it safe twice, once at the reservation and once at the destination.

One scope note before anything else. The experiments here are threads on one machine with real clock leases, and the "external system" is an object guarded by a thread lock. This is a simulation of a distributed system, not a distributed system, and I will not claim otherwise. What the simulation gives you is exact, repeatable timing evidence for the state machines involved, which is the part most explanations wave their hands at.

What the SDK already gives you

Two threads invoked the same Agent instance with different prompts. One call won. The other raised: ConcurrencyException: Agent is already processing a request. Concurrent invocations are not supported.

That is the documented default. The constructor argument concurrent_invocation_mode defaults to THROW, and UNSAFE_REENTRANT is the explicit opt-out for code that accepts the risk:

# strands/agent/agent.py:268 -- as installed, strands-agents 1.63.0
concurrent_invocation_mode : ConcurrentInvocationMode = ConcurrentInvocationMode . THROW ,
# strands/types/agent.py:205
class ConcurrentInvocationMode ( str , Enum ):
  THROW = " throw "
  UNSAFE_REENTRANT = " unsafe_reentrant "

Line numbers move with every release, so every SDK excerpt in this post is pasted from the installed 1.63.0 instead of being left at the end of a link. The link points at the file, not at a line, and the excerpt is the thing I actually tested against.

The second probe was more interesting. The call signature accepts an idempotency_token, a caller-chosen name for one logical request, so a retry carries the same name as the original instead of looking like a second request:

# strands/agent/agent.py:924-929 -- keyword-only arguments of __call__
idempotency_token : Any = None ,
limits : Limits | None = None ,

When a second thread called the same instance with the same prompt and the same token, the duplicate did not crash and did not run the model twice. It waited for the inflight invocation and received its cached result. Both threads reported identical usage, 948 total tokens each, because only one invocation ever ran.

One branch of one method decides that:

# strands/agent/_concurrency.py:183
def _check_idempotency ( self , idempotency_token : Any ) -> tuple [ _InflightInvocation | None , Any ]:
  if idempotency_token is None or self . _mode != ConcurrentInvocationMode . THROW :
    return None , None
  with self . _inflight_lock :
    if self . _inflight_token == idempotency_token :
      return self . _inflight , None

A second call that matches on == gets the inflight invocation handed back to it, then waits on that object rather than starting a second model request. So the SDK already distinguishes "concurrent invocations" from "the same request twice." It handles the second one correctly.

The third probe found the boundary. Two separate instances, one thread each, each invocation carrying limits={"turns": 1}, both ran to stop_reason="limit_turns" with identical usage of 453 total tokens each. No budget was shared between them. None could be, because limits travel with the request, not with the fleet:

# strands/agent/agent.py:956-957 -- docstring of __call__
limits : Limits for the whole request this call starts ( turns / output_tokens / total_tokens / structured_output_attempts ).

The two usage figures differ for an unrelated reason. The token-dedup probe ran to end_turn over two turns. The limits probe stopped after one turn.

In short, the SDK serializes an instance, dedupes a repeated token in-process and budgets each call separately. That guard, the token registry and the lock all live in one per-instance controller, which is why a second process cannot reuse them without inventing a protocol of its own:

# strands/agent/_concurrency.py:95 -- constructed as self._concurrency in Agent.__init__
def __init__ ( self , mode : ConcurrentInvocationMode ) -> None :
  self . _mode = mode
  self . _invocation_lock = threading . Lock ()
  self . _inflight_token : Any = None
  self . _inflight : _InflightInvocation | None = None
  self . _inflight_lock = threading . Lock ()

Every piece of that state hangs off self and no module-level registry sits behind it, so two processes share nothing. A quota that several workers must share is still yours to build, which is where leases come in and where the trouble starts.

What a lease can and cannot tell you

The minimal system has one capacity slot, a unit of the shared quota that article 03 bounded per call, one worker that reserves it and one external system, a ticket service, that records side effects. The worker reserves the slot, performs the effect and confirms. The reservation carries a lease, a deadline after which the slot may be taken back, and a janitor loop watches that deadline on the real clock.

A lease is therefore a one-sided promise, because it protects the fleet from a silent worker and it can also take a slot away from a worker that is still healthy. The lease carries no heartbeat and no owner record. In the experiment it is three methods on one object, and the whole of it is a deadline on the monotonic clock:

# src/lease_crash_boundary.py:56
def reserve ( self , lease_seconds : float ) -> None :
  with self . lock :
    self . deadline = time . monotonic () + lease_seconds
def expired ( self ) -> bool :
  with self . lock :
    return self . deadline is not None and time . monotonic () >= self . deadline
def release ( self ) -> None :
  with self . lock :
    self . deadline = None

The naive policy treats expiry as proof of innocence: the slot is free, so the operation never happened, so a retry worker re-executes the job. I crashed the worker after the effect had landed but before the confirmation and ran the race five times.

POLICY=naive CRASH=after-effect trial 1: expiry@1.0s effect@0.205s retry@1.204s effects=2 duplicates=1
trial 2: expiry@1.002s effect@0.202s retry@1.206s effects=2 duplicates=1
trial 5: expiry@1.009s effect@0.201s retry@1.214s effects=2 duplicates=1
SUMMARY trials=5 duplicate_writes_total=5 capacity_returned_while_effect_exists=5/5

Every run wrote the ticket twice. Every run handed the capacity back while a completed effect was sitting in the external system. Scale to many workers and that freed slot goes to an unrelated job, so the fleet quietly spends two units of quota on one.

The lease told the truth, just not the truth the policy needed. "The worker stopped reporting" is a fact about the worker. "The operation did not happen" is a fact about the external system, which a lease cannot see.

The reconcile policy separates the two facts. On expiry it holds the reservation in an outcome_unknown state and asks the external system whether the idempotency key exists before touching capacity. When the crash lands after the effect, the lookup finds the ticket, so the reservation settles as confirmed_existing and no retry runs. When the crash lands before the effect, the lookup finds nothing, so the reservation settles as unused and the capacity returns. The retry that follows is correct, because there was nothing to duplicate. Both crash points, one policy, no damage; all ten runs sit in evidence/lease-crash-boundary.txt.

The distance between the two policies is one branch of the janitor loop:

# src/lease_crash_boundary.py:99 -- runs once expiry has been observed
if policy == " naive " :
  state = " unused " # expiry treated as proof the effect never landed
  slot . release ()
  retried = True # retry thread: sleep RETRY_LATENCY, then dest.execute(key)
else :
  if dest . has_effect ( key ):
    state = " confirmed_existing "
  else :
    state = " unused " # the only path back to capacity
    slot . release ()
    retried = True # the same retry thread

The naive branch never asks the destination anything, and that one omission is the difference between five duplicate writes and none.

Neither policy fixes the cost that a crash charges to everyone else, and the contention experiment, this time with two slots instead of one, makes that cost visible. Twelve worker threads contended for two leased slots with a one second lease. Worker 0 crashed while holding one of them. That slot came back only through expiry, and the worker that inherited it waited 1.002 seconds, almost exactly the lease length, before it could start. The queue behind a crash pays for the crash even though nothing is running on the held slot.

Two states, not one

The fix above looks like a policy flag, but it is a state machine change. The naive policy has no intermediate state: expiry sends reserved straight to unused. The reconcile policy inserts outcome_unknown between them and requires evidence to leave it.

naive:reserved ──expiry──▶ unused
reconcile:reserved ──expiry──▶ outcome_unknown ──key found──▶ confirmed_existing
├──key absent─────────▶ unused
└──attempt in flight──▶ outcome_unknown (hold)

The outcome_unknown state is the whole game. In other words, the reservation system now knows something about the worker (it stopped) and nothing about the operation. Capacity only returns through one of the two exits, both of which require evidence from the external system rather than evidence from the clock.

The retry that arrives too early

A careful reader spots the hole in the reconcile policy. It is the same reader who spotted it in the comments section. The lookup answers a question about a moment in time: does the key exist right now? The old attempt is still in flight when the lookup runs, and its commit can land after the answer. Absence at lookup time is not absence at commit time.

The deterministic probes in the comment thread simulated this with a logical clock. The upgrade in this post is a real one. The old attempt runs in a real thread with a real 2.4 second workload. The lease expires at 1.0 second on the monotonic clock. The janitor loop genuinely reconciles at that moment, while the old thread is still sleeping (the first harness prints POLICY=naive, the second prints MODE=naive, and both are the same policy in two shapes).

Five runs of the naive reconciler:

MODE=naive trial 1: expiry@1.003s released@1.003s old_commit@2.405s(ok=True) retry@1.407s effects=2 duplicates=1
trial 2: expiry@1.001s released@1.001s old_commit@2.404s(ok=True) retry@1.405s effects=2 duplicates=1
SUMMARY trials=5 duplicate_writes_total=5 early_capacity_release=5/5

The janitor loop released capacity at 1.003 seconds. The retry committed at 1.407 seconds. The old attempt, still alive this whole time, committed at 2.405 seconds. Two writes, one ticket. The state machine never noticed, because both of its decisions were locally correct at the moments it made them.

The reconcile policy from the previous section does not survive contact with a slow attempt. The retry here is a stand-in for any other worker that picks up the same job after the release, and a released job has nothing that stops a second pickup. If your workers share one permanent identity instead of per-attempt ids, the same ambiguity returns through a different door: a crashed attempt and a slow attempt are indistinguishable under one id. Cancelling the id on death is a separate mechanism, and this experiment does not cover it.

Fences and witnesses

Two disjoint repairs exist. They fix different halves of the damage. The first repair is a uniqueness check at the destination, a put-if-absent on the idempotency key. I will call it a fence, with one honest caveat: it is not a fencing token, because it does not order attempts. It refuses whichever write arrives second, and it cannot stop an old write that arrives first.

The check sits in the destination's commit path, not in the reservation:

# src/delayed_commit_race.py:46
def commit ( self , attempt_id : str , key : str ) -> bool :
  with self . lock :
    if self . fenced and key in self . effects :
      self . rejected += 1
      self . attempts [ attempt_id ] = " fenced "
      return False
    self . effects . append ( key )
    self . attempts [ attempt_id ] = " committed "
    return True

MODE=fenced trial 1: expiry@1.006s released@1.006s old_commit@2.407s(ok=False) retry@1.409s effects=1 duplicates=0
SUMMARY trials=5 duplicate_writes_total=0 early_capacity_release=5/5

In these runs the fast retry committed first at 1.409 seconds, so the old attempt was the one turned away at 2.407 seconds after burning its full workload. Had the old attempt arrived first, the retry would have been the refused one instead. Either order ends with one write. The fence does not care which attempt loses.

Zero duplicate writes across five runs, so the ticket system is safe. But look at what the fence did not prevent. Capacity was still released early in all five runs. The retry still ran. The losing attempt spent its whole workload only to be told no. As a result, the fence protects the destination, but not the capacity or the wasted effort.

In contrast, the second repair is a destination that witnesses attempt state. If the external system exposes that attempt K is in_flight, the janitor loop does not have to guess from a key lookup. It asks two questions inst