Hot-loading a regex word list from a website into a production API without getting ReDoS'd

By Jordan Tate, VP of Customer Relations at Conduyt.

We run a free message checker on our website. You paste a text message or an email draft, and it flags words that carriers and inbox providers tend to filter, with a reason and a safer swap for each one. The same engine sits inside our CRM's API, where it checks drafts before they go out.

The engine grades a draft against two word lists: one for email, one for SMS. Each entry is a regular expression plus a category, a level for each channel, a short explanation and a list of replacements. The lists change more often than the engine does. Our marketing side re-reads carrier and provider sources every week and updates the lists on the website.

For a while that meant two copies: the website's lists, and the copies bundled into the API, which only moved with a release. So we made the API read the website's lists directly. A word change on the website now reaches the API within the hour, with no release.

That raised the obvious question. A file that marketing can edit, full of regexes, is now compiled and run on every API request. How do you make sure it can never take the API down or slow it? This post is the answer we shipped. Nothing in it is specific to spam words. It applies to any config file you accept from a source you do not fully control.

The shape of the problem

There are three ways a fetched list can hurt you:

• It is not really a list. The website serves the lists as JavaScript files, because the browser checker loads them with a script tag. If the server ever ran that file, anyone who could edit it could run code on our API.
• It is a list, but a pattern in it backtracks catastrophically. One bad regex, such as a nested quantifier, can pin a CPU for seconds on a crafted input. That is ReDoS.
• It is a list, every pattern is fine on paper, and it still runs slow in production, because a static check can be wrong.

We handle each of these in a separate layer, and every layer fails closed.

Layer 1: fetch rarely, keep a fallback

The API fetches the two files at most once per hour per server instance. A failed fetch waits five minutes before the next attempt, and each file gets a five-second timeout for the whole exchange.

The API still ships with bundled copies of both lists, pinned by md5. On a cold start where the fetch fails, or where the fetched file is refused, requests are answered from the bundled lists. A refresh that fails later keeps the last good copy. Whatever happens on the website, the API always has a list to check against.

Layer 2: parse as data, never as code

The fetched text is data. Nothing fetched is executed: no eval, no Function, no import. Instead, a strict hand-written extractor reads the one assignment statement the list files are made of.

A list file looks like this:

(typeof window !== "undefined" ? window : globalThis).SPAM_SMS = [
  ["pattern", "category", "smsLevel", "emailLevel", "why", ["swap", "swap"]],
  ...
];

The extractor accepts block comments and whitespace, then exactly that head (or its minified form), then a literal made of arrays and single or double quoted strings, nested at most three deep, then an optional semicolon, then the end of the file.

Anything else refuses the whole file with a logged reason: a function, a second statement, an identifier, a number, an object, a trailing comma, an unclosed string, a body that is too big.

After parsing, every entry's shape is checked: six or seven fields, all strings except the replacements array, levels from a fixed set, and a pattern that compiles.

There are hard caps before any of the expensive work starts:

• 512 KB per file. The bundled files are 54 KB and 29 KB.
• 2,000 entries per file. The bundled lists have 382 and 214.
• 500 characters per pattern.

If either file is refused, the lists in use do not change.

Layer 3: analyse every regex for super-linear behavior

This is the ReDoS layer. Every pattern goes through two analysers, and one refused pattern refuses the file.

The first is scslre, a static analyser for super-linear backtracking. We do not run it on the bare pattern. We run it on the two regexes the engine actually builds from each pattern: the scanner, which wraps the pattern in word-boundary groups and runs with the global flag over the whole draft, and the whole-word test, which anchors it. The regex that runs is the one that can backtrack, so that is the one we check. Any report from scslre refuses the pattern.

The second is an automaton check. It builds the pattern's position automaton and looks for exponential and polynomial degree of ambiguity, which catches structural ambiguity that the first analyser cannot see.

Before either analyser runs, the pattern must parse the way the engine compiles it, pass structural caps, contain no backreference, and follow one lookaround rule: a lookaround body may not repeat without bound. The one exception is a lookahead whose unbounded repeat is whitespace only, because a run of spaces ends at the next non-space character.

Here is what that looks like for two patterns, run through the same check the API uses:

((a+))+$ -> refused: pattern_superlinear_self (2.1 ms)
exam\s+dumps? -> accepted (1.5 ms)

The first is a classic nested quantifier, and it is one of the cases in our test suite. The second is the shape of an ordinary list entry: literal words, a whitespace run, an optional letter.

Layer 4: budget the validation itself

An analyser is code too, and it can be slow on some input nobody has tried yet. If validation can stall, an attacker who can edit the list has just moved the ReDoS from the regex into the validator. So the validation is bounded and fails closed:

• Structural caps run first, before any analyser builds anything.
• One work counter covers every phase of the automaton checks. When it runs out, the pattern is refused as too complex.
• Each pattern gets a 50 ms wall-clock budget, enforced with a node:vm timeout. A pattern that hits the clock is tried once more, because a garbage collection pause on a busy machine can stretch a normal run, and is refused if it times out again.
• The whole refresh gets a budget of patterns times 50 ms, capped at 10 seconds. Past that deadline the file is refused.

A note on node:vm: here it is a timer, not a sandbox. It runs one fixed line of our own code that calls our own function. The fetched text never goes into it. We use it because V8's watchdog can interrupt a running regex, which a plain setTimeout cannot.

The bundled patterns peak at about 5 ms each, so the 50 ms budget leaves room without letting anything run long.

Layer 5: budget the runtime, and quarantine what times out

A list that passed every check is still not trusted blindly. A static check can be wrong, and the analysers are static checks. So every call into an engine built from the remote lists runs under a wall-clock budget, 250 ms by default, again through a node:vm timeout.

If a request runs past it, three things happen:

1. The request is answered by the bundled engine, so the caller still gets a result.
2. The exact md5 pair of the two list files is quarantined for that server instance. Every later refresh that fetches the same pair refuses it, and a remote engine already handed to another request is checked against the quarantine right before it runs.
3. The next refresh is moved up to the five-minute retry, not the hour, because the website may be fixed soon.

Quarantining by content hash rather than by URL means a fixed file gets a fresh chance as soon as its bytes change, and a broken one never gets a second run on that instance.

The bundled engine is our own reviewed code and runs unguarded. The browser composers inside the CRM also keep the bundled engine. There is no client-side fetch of the lists at all, so this whole path lives on the server.

Layer 6: log every refusal with a reason

Every refusal logs one shape: the file, the host, the reason and which source was in use. The reasons are short machine-readable strings, such as oversize, trailing_comma with the character offset, entry_214_pattern_superlinear_self, wordlist_validation_timeout, quarantined_engine_timeout or engine_timeout_250.

That matters for the people editing the lists. When a weekly update is refused, the log names the entry and the rule it broke, and the fix is usually one line.

What a weekly update looks like

This week's re-read of Twilio's forbidden message categories and Telnyx's forbidden use cases added five SMS rows, taking that list from 214 to 219. The new rows cover initial coin offerings, simulated phishing and exam dumps, plus ghostwriting and vaporizers at caution level. Each new pattern passed both analysers in under 10 ms.

A sixth candidate, guaranteed returns on investments, was withdrawn during review: once you exclude agricultural yields, productivity gains, retail returns policies and clause-length disclaimers, the claim cannot be expressed safely as a list row, so it goes back to the engine as a feature request rather than shipping as a regex.

The files are served under content-hashed names, using the first eight hex characters of the file's md5. An independent Codex review runs on the change before it is deployed. After that, the API picks the lists up on its next hourly refresh, with no release.

You can try the result in our text message checker and email spam checker, and the reasoning behind the words is in our spam trigger words guide.

The checklist

If you accept a config file you do not fully control, and especially one full of regexes:

• Parse it as data. Write a strict reader for the exact format, and refuse everything else.
• Cap sizes before any expensive work: bytes, entries, pattern length, nesting depth.
• Analyse every regex for super-linear behavior, and analyse the regex you actually run, not the fragment.
• Budget the validation, per item and in total, and treat a timeout as a refusal.
• Budget the runtime, because static checks can be wrong.
• Keep a known-good fallback that does not depend on the remote source.
• Quarantine what times out, by content hash.
• Log every refusal with a reason a person can act on.

None of this is clever. It is a set of small fences, each one assuming the one before it might fail. The result is that the people who know the words can change them every week, and the people who run the API do not have to worry about it.

Drafted with AI assistance from Conduyt's message-check source code; constants and behavior checked against the server module that loads the lists.