DEV Community 1h ago

Timeout is not failure: the email bug every AI writes

I build back-office systems for small Italian companies: CRMs, booking flows, the boring machinery that sends the right email to the right person. I write most of that code with an AI assistant. I'm not hiding it, it's how I work.

And there is one bug my assistant writes almost every time, in every project, unless I stop it. It looks like good engineering. It passes the tests. And in production it sends your customer the same email twice.

The code that looks right

Ask an AI to "send the confirmation email, and make it robust", and you'll get something like this:

async function sendConfirmation(booking: Booking) {
  for (let attempt = 1; attempt <= 3; attempt++) {
    try {
      await emailProvider.send(confirmationFor(booking));
      return;
    } catch (err) {
      if (attempt === 3) throw err;
      await sleep(1000 * attempt);
    }
  }
}

Retries, backoff, a final throw. A reviewer would approve it. So would I, on a tired day.

The problem is hidden in that catch. It treats every error the same way: the email didn't go out, try again. That's true for some errors. For the most common one in production, it's not.

What a timeout actually tells you

Say the request to your email provider times out after 10 seconds. What do you know? Not much. Maybe the request never left your server. Maybe it reached the provider, the provider queued the email, and the response got lost on the way back. Maybe the provider is just slow today and your email is already in the customer's inbox.

A timeout doesn't mean "failed". It means "I don't know".

This isn't news: it's one of the first things Martin Kleppmann explains about unreliable networks in Designing Data-Intensive Applications. When you get no answer, you can't tell a lost request from a lost response. But it's easy to forget when the code that handles it is three lines long and written by someone else, or by something else.

And when you retry on "I don't know", you sometimes send it twice. For a newsletter, that's embarrassing. For a booking confirmation, a password reset, an invoice or a medical report, it's a support ticket, or worse, a customer who stops trusting your system.

The tests don't catch it because the fake email provider in your tests either succeeds or fails cleanly. Real networks have a third state, and nobody wrote a test for it.

First, the real fix (if you can have it)

If your provider supports an idempotency key, use it. You send a unique key with each email, the provider remembers it, and a retry with the same key is ignored instead of sent again.

That's how payment APIs make retries safe (Brandur Leach's write-up of Stripe-style idempotency keys is the classic explanation), and it's the answer Malcolm Featonby gives in the Amazon Builders' Library: make the write idempotent, then retry freely.

Many transactional email APIs don't offer that. The one I use doesn't document one. So the rest of this post is what you do when the other side can't deduplicate for you.

Three outcomes, not two

Sending an email then has three possible results:

Sent. The provider answered "OK".
Refused. You know for sure it didn't go out: the provider rejected the request, or the request never reached it. Safe to retry.
Uncertain. You asked and you got no clear answer. Never retry automatically.

In code, I make the third one a type of its own, so nobody (human or AI) can treat it like the others by accident:

/** The provider rejected it, or the request never reached it: the email did not go out. Safe to retry. */
export class DeliveryRefused extends Error {
  override readonly name = 'DeliveryRefused';
}

/** We don't know if the email went out: no answer in time, or the connection broke after the request was sent. */
export class DeliveryUncertain extends Error {
  override readonly name = 'DeliveryUncertain';
}

The interesting part is deciding which is which. My rule: an error is "refused" only if it proves the email didn't go out. On the network side, that means the request never arrived: the DNS name didn't resolve, nobody was listening on the port, the network had no route, the connection was never opened. On the HTTP side, it means the provider looked at the request and said no (a 4xx).

Everything else is uncertain, the timeout first of all, and a 5xx too: a gateway that answers 502 or 504 may have passed your request on before giving up.

// network errors that prove the request never reached the provider
const NEVER_ARRIVED = new Set([
  'ENOTFOUND',
  'EAI_AGAIN',
  'ECONNREFUSED',
  'EHOSTUNREACH',
  'ENETUNREACH',
  'UND_ERR_CONNECT_TIMEOUT',
]);

// the email adapter: the only place that knows how the provider talks
async function sendEmail(msg: Message, config: EmailConfig): Promise<void> {
  let res: Response;
  try {
    res = await fetch(config.providerUrl, {
      method: 'POST',
      body: JSON.stringify(msg),
      signal: AbortSignal.timeout(config.timeoutMs),
    });
  } catch (err) {
    const code = networkErrorCode(err); // reads err.code or err.cause.code
    if (code && NEVER_ARRIVED.has(code)) throw new DeliveryRefused(`never arrived (${code})`);
    throw new DeliveryUncertain('no answer, the email may have gone out');
  }
  if (res.status >= 500) throw new DeliveryUncertain(`provider answered ${res.status}, it may have gone out`);
  if (!res.ok) throw new DeliveryRefused(`provider rejected it (${res.status})`);
}

Notice the default. When in doubt, the code says "uncertain". An AI tends to write the opposite: it lists the scary errors and treats everything else as "retry". The safe default has to be written down, because it's not the one that comes naturally.

(Reads are different. If a request that only reads times out, nothing changed on the other side, so you can retry it, with backoff, a little random jitter and a limit, as the Google SRE book and Marc Brooker's piece on backoff and jitter both recommend, so your retries don't pile onto a provider that is already struggling. The rule here is only about calls that write: send an email, charge a card, post a message.)

Claim it before you send it

Three outcomes are not enough on their own. If your server restarts in the middle of a send, the retry logic is gone, and a background job might pick up the "unsent" email and send it again.

So before sending, the code claims the email in the database. A claim is a row that says "this email is being sent, hands off". It's written and committed on its own, and the send happens after, with no database transaction left open while we wait for the provider.

The send happens only if the claim succeeds, and what happens next depends on the outcome:

type SendOutcome = 'sent' | 'already-claimed' | 'uncertain';

async function sendEmailOnce(id: string, send: () => Promise<void>): Promise<SendOutcome> {
  if (!(await claimEmail(id))) return 'already-claimed'; // someone else has it, or it's already done
  try {
    await send();
  } catch (err) {
    if (err instanceof DeliveryUncertain) {
      alertHuman(`email ${id}: outcome unknown, stays claimed, will NOT be resent`);
      return 'uncertain'; // the claim stays: nobody resends it
    }
    await releaseClaim(id); // only a sure "no" frees the claim
    throw err;
  }
  await markEmailSent(id).catch(() =>
    alertHuman(`email ${id} went out but the mark failed: stays claimed, will not be resent`),
  );
  return 'sent';
}

Read the branches again, because each one is a decision:

Refused frees the claim, so the next run of the job retries it. Correct: we know it didn't go out.
Uncertain keeps the claim and tells a person. The email is stuck on purpose. One stuck email a month is cheap; one duplicate medical report is not.
Sent but not marked (the database write failed right after a successful send) also keeps the claim. Again: never resend.

The process dies between the claim and the answer. Nobody is left to alert anyone, so the recovery job does it: a claim with no outcome, older than a few minutes, is treated as uncertain. It gets flagged, never resent.

In the vocabulary of distributed systems, this is choosing at-most-once delivery: an email may occasionally need a human, but it never goes out twice. The opposite choice, at-least-once (the transactional outbox pattern that Chris Richardson describes in Microservices Patterns, for example), never loses a message but needs the receiver to deduplicate. Without an idempotency key, the receiver here is your customer's inbox, and inboxes don't deduplicate.

Two details that make the human part cheaper:

Ask the provider later. If your provider has an API for message events or logs, the recovery job can look up the uncertain email after a while and settle it on its own: found means sent, so mark it. Only what's still unknown reaches a person.
No apology on uncertainty. When the outcome is uncertain, my flows also skip the "sorry, something went wrong" email. If the customer may already have the real one, an apology for a failure that maybe didn't happen is just a second confusing email.

And then you lock it in

Here's the part that matters most when an AI writes your code: a rule in a comment is a rule the AI will forget. Next month I'll ask for a new email flow, the assistant will write the friendly retry loop again, and it will look fine. So the rule becomes a test that fails.

The fake email provider in my tests can do the nasty thing real networks do: accept the email, then time out.

it('never resends an email whose outcome is uncertain', async () => {
  const provider = fakeProviderThatAcceptsThenTimesOut();
  await sendConfirmation(booking, { provider });
  await runRecoveryJob({ provider }); // the job that picks up "unsent" emails
  await sendConfirmation(booking, { provider }); // and a user clicking twice
  expect(provider.accepted).toHaveLength(1); // one email reached the customer, not three
});

This test is the real guardrail. When a future change brings back the naive retry, the test goes red, and the AI has to explain itself. That's a pattern I use everywhere now: when I find a trap that matters, it doesn't go into a note. It goes into a check that blocks.

The rule, in one line

A read can be retried. A write can be retried only if it's idempotent, or if you're sure it never arrived. It applies far beyond email: payments, SMS, WhatsApp messages, webhooks you call, anything that changes the world outside your database.

Your AI assistant knows how to write a retry loop. What it doesn't know is which of your errors mean "I don't know". You have to tell it, and then you have to make the test suite remember.