Sequential Retries Pass, Concurrent Duplicates Fail: Black-Box Testing for Idempotency-Key Races

Problem Overview

Sequential retries can pass while concurrent requests sharing one Idempotency-Key still produce more than one logical result. A retry loop sends a request, waits for the response, then sends the next one, so the second request finds the stored result. A burst does not wait, and both requests do the work simultaneously.

Sequential retries: PASS
Concurrent duplicates: FAIL
Same Idempotency-Key: ??multiple logical results

The typical implementation follows this pattern:

check key ??do work ??create resource ??store result

Two concurrent requests pass the check before either stores its result. Both create a resource, and the caller ends up with two order_id values for one intended operation.

What IdemCheck Does

IdemCheck is a small Go CLI for black-box HTTP testing. It sends requests and compares responses - something you have when the API belongs to a partner or vendor. The tool runs same-key concurrency on purpose, then replays the original request and checks whether everything converged on one observable result. The comparison covers status codes and JSON structure, and the report names the field that diverged, for example $.order_id.

Core Workflow

• BURST - releases ten requests sharing one key, then waits 250ms (SETTLE), followed by REPLAY (sending the original request again).
• VERDICT - reports whether the responses converged, mapping to exit codes: PASS → 0, FAIL → 1, ERROR → 2, INCONCLUSIVE → 3.

Ten concurrent requests on one key is the default concurrency level.

Test Scenarios

Step	Action
--trials	Repeats the burst with a fresh key when the race window is narrow
--concurrency	Raises the number of concurrent requests
--allow-remote	Prints a warning before sending duplicates to remote targets

This run uses two demo servers in the repository: the unsafe server checks the key, sleeps, inserts, then stores it, while the safe server locks per key.

Installation and Usage

Install the tool via Go:

go install github.com/hyukvoid/idemcheck/cmd/idemcheck@v1.0.0

Point it at an authorized environment:

idemcheck test \
  --url https://staging.example.com/orders \
  --body-file request.json \
  -H "Authorization: Bearer $TOKEN" \
  --allow-remote

Loopback targets work without flags. Remote targets require --allow-remote, which prints a warning before sending duplicates.

Curls, k6, and vegeta can also send these requests and report latency. IdemCheck reads the responses under one key and decides whether they agree.

Honest Limitations

A one-off script can test a simple case, but IdemCheck exists to package the same-key burst, replay, response comparison, verdicts, and CI behavior together. The checks are written once and run on every endpoint. A PASS means no violation was observed through HTTP. It does not prove hidden database writes, messages, emails, payment captures, or other internal side effects happened exactly once.

Feedback

We welcome input from engineers working with payments, orders, webhooks, background jobs, and retry-heavy APIs. Has this matched the races you've actually hit? What does a run like this miss on your endpoints? If a verdict disagrees with what you see in production, that gap is the useful bug report. Open an issue or a PR: https://github.com/hyukvoid/idemcheck