DEV Community

Our thirty second timeout let one call to the customs portal run for forty minutes

Every evening a Python worker submits our export declarations to our customs broker's API, one consignment at a time, so that the lorries leaving at six the next morning have their clearance.

Years ago a hung partner call had taken down a service of ours, and since then every outbound client we own has a timeout. This one used requests with timeout=30. We read that as: no call can take longer than thirty seconds.

A bad night at the broker

In October the broker's platform had a bad night. Their gateway stayed up, but responses came back a few kilobytes at a time. One submission returned a large validation report, and its call ran for forty one minutes without raising anything.

The worker processes declarations in order, so the three hundred behind it waited. Ninety consignments missed the morning lorries.

What timeout actually means

The requests documentation is precise about this, and we had never read it. The timeout is not a limit on the whole response. It applies to connecting, and then to each wait for data on the socket: the exception is raised only if no bytes have arrived for thirty seconds.

A server that sends something every twenty nine seconds can keep a call open for as long as it likes. A slow trickle never trips a timeout that only measures silence.

Our dashboard did not help either. We recorded the duration of calls that completed and the number that timed out. A call that never finished appeared in neither, so for forty minutes the graphs showed a quiet evening.

Enforcing a total deadline

The worker now enforces a total deadline per call. It streams the response and checks the elapsed time on every chunk, abandoning the call at sixty seconds whatever the server is doing, and the submission goes to a retry queue instead of blocking the run.

Declarations are submitted by four workers in parallel, so one stuck call delays a quarter of the run at worst. We also emit a metric for calls in flight with their age, and an alert fires when any outbound call has been open for longer than twice its deadline.

We audited the other clients and found the same assumption in eleven of them. The library's name for the setting and our idea of what it meant were the same word for two different things.

  • Sergey Shinder
Read on DEV Community ↗ ← Back to News

Comments

No comments yet. Start the discussion.