The new customer who took down everyone: an ORM include that ate our CRM

My entry for DEV's Summer Bug Smash - Smash Stories. A real production war story about how one endpoint, one include too deep, and one very enthusiastic new customer combined into a global outage - and what actually fixed it (after the first fix didn't).

The 15 minutes where everything was down

It started the way these things always do: a new customer onboarded, we imported their data, and minutes later everyone's app started falling over. Not just the new customer - the whole multi-tenant CRM. Dashboards spun. Deal pages hung. Then requests started timing out across the board.

The trigger was data volume. The new account came in with a lot of history - far more opportunities, contacts, activities, and line items than our typical tenant. Nothing about that should take down other customers… except it did.

Following the smoke

The outage lined up almost perfectly with traffic to one endpoint: the one that returns the full details of an opportunity/deal. On paper it's a read. In practice it had grown, over a couple of years, into a monster.

It was a single Sequelize query with a tree of includes:

Opportunity.findByPk(id, {
  include: [
    { model: Account, include: [Contact, Address] },
    { model: LineItem, include: [Product, { model: PriceTier, include: [Currency] }] },
    { model: Activity, include: [User, Note] },
    { model: Stage, include: [Pipeline] },
    // …and more
  ],
})

Every include became a JOIN (or a follow-up query), and several of them pulled in their own nested includes. For a normal-sized deal it was fine. For the new customer's fattest opportunities - hundreds of line items, each with products and price tiers, plus a long activity history - a single "show me this deal" call was fanning out into an enormous result set and a pile of un-indexed lookups.

The query blew past our HTTP timeout, the request died, the client retried, and the retries piled onto an already-struggling database. Connections backed up, and because every tenant shared the same pool, everyone felt it.

Fix #1: the one that felt good and wasn't

Under pressure, we did the obvious thing: we raised the timeout. The endpoint stopped "failing," the alarms went quiet, and we told ourselves we'd bought time to do it properly.

That was a band-aid on a severed artery. All we'd done was let the slow query run longer while holding a connection hostage the entire time. A week or two later, with a bit more data and a bit more concurrency, the exact same failure came back - except now each hung request lingered even longer before dying, so the pile-up was worse.

Raising a timeout doesn't make a query fast; it just changes how long you wait to find out it's slow.

Fix #2: the one that actually held

The real fix had two halves.

1. Stop returning everything at once. We split the single mega-endpoint into several focused calls. The deal page didn't actually need every nested activity, note, and price tier on first paint - it needed the core opportunity, and then the related collections lazily, as the user scrolled to them. So we replaced the one-query-to-rule-them-all with:

   • a lean GET /opportunity/:id that returns just the opportunity and its immediate summary, and
   • separate, paginated endpoints for the heavy children (/opportunity/:id/line-items, /activities, etc.), each returning only the columns the UI renders.

   That alone collapsed the worst-case payload from "the entire deal graph" to something bounded and predictable, and it meant a single fat deal could no longer monopolize a connection for seconds.

2. Index the paths we actually query. The child lookups were hitting foreign keys and filter columns that weren't indexed, so they degraded from index seeks to sequential scans as the tables grew - which is exactly why the problem only showed up with a large tenant. We added the missing indexes (the opportunity_id foreign keys on the child tables, plus the columns we filtered and sorted on) so those per-collection queries stayed fast regardless of how much history a customer had.

Split the work, return only what's needed, and index the access paths - and the outage stopped recurring. We haven't seen it since. (I'm knocking on wood as I type this.)

What I took away from it

• A timeout increase is not a fix. It's a way to postpone the same failure into a worse one. If raising a limit makes the symptom disappear, the real bug is still there - you've just muted the alarm.
• include depth is a loaded gun in any ORM. Every convenient nested include is a JOIN or an N+1 waiting to matter. It's invisible at small scale and catastrophic at large scale.
• In multi-tenant systems, one customer's data volume is everyone's problem when they share a connection pool. "It's slow for one account" quietly means "it's fragile for all of them."
• Slowness usually lives at the access path. The endpoint looked like an application problem; the fix was mostly shape of the data returned + indexes. Measure before you rewrite.

The unglamorous version of the lesson: return less, and make sure the database can find what you ask for. Both halves mattered - decomposition kept any one request bounded, and indexing kept each of those smaller requests fast as the data grew.