Retry Storm Lab

Comments

A retry is not free. See when your recovery policy becomes the outage.

Overview

Retry Storm Lab is a deterministic Python simulator for partial dependency failures. It models fresh requests and retries competing for the same capacity, then compares immediate, fixed, exponential, and full-jitter backoff policies.

Retries improve reliability when failures are brief and spare capacity exists. During a sustained partial outage, synchronized clients can instead amplify demand, starve fresh requests, and keep a recovering service overloaded. The simulator makes that feedback loop visible:

• Generate a steady stream of fresh requests.
• Reduce successful capacity during a configurable failure window.
• Reschedule failed and over-capacity attempts using the selected policy.
• Measure amplification, peak demand, abandoned requests, and recovery time.
• Compare all four policies against the exact same scenario and random seed.

This is an intentionally small queueing model, not a production capacity planner. It omits network latency distributions, client timeouts, autoscaling, circuit breakers, and correlated downstream failures. Its job is to make retry dynamics easy to inspect and discuss.

Requirements

Python 3.11 or newer is required.

Setup & Run

python -m venv .venv
source .venv/bin/activate
pip install -e .
streamlit run app.py

Open http://localhost:8501, choose a preset, and adjust the workload from the sidebar.

CLI Usage

retry-storm-lab --preset "Rate-limit spiral" --compare
retry-storm-lab --preset "Slow recovery" --policy full_jitter --json

Example Comparison

immediate     amplification=1.11x  peak=2720 rps  success=97.2%  recovery=1s
fixed         amplification=1.08x  peak=1600 rps  success=99.9%  recovery=3s
exponential   amplification=1.07x  peak=1600 rps  success=100.0% recovery=3s
full_jitter   amplification=1.10x  peak=1651 rps  success=99.9%  recovery=3s

Exact values depend on the selected preset and seed.

Python API

from retry_storm_lab import BackoffPolicy, SimulationConfig, simulate

result = simulate(
    SimulationConfig(
        baseline_rps=800,
        server_capacity_rps=1_000,
        outage_failure_percent=80,
        policy=BackoffPolicy.FULL_JITTER,
    )
)

print(result.amplification_factor)
print(result.peak_rps)
print(result.recovery_seconds)

Technical Details

The engine uses aggregate 100 ms buckets. Full jitter distributes retries individually with a seeded pseudo-random generator, so the same configuration always produces the same result.

The lab works entirely offline. To record aggregate usage bands with Telemetry, install the optional dependency and provide a project API key:

pip install -e '.[telemetry]'
export TELEMETRY_API_KEY=your_project_api_key
streamlit run app.py

Only the selected policy, broad capacity/amplification bands, and whether the run recovered are sent. Raw workload values and timeline data are not logged.

Testing

python -m unittest discover -s tests -v
python -m compileall app.py retry_storm_lab

License

MIT