Six AI agents froze my production server. Each one did the right thing

This article was created with the help of AI. The incident, the logs and the numbers are from my own server.

On September 25 at 19:01 UTC my server stopped responding. The site, the admin panel and the publishing queue all froze at once. Seventeen minutes later I hard-rebooted it. There was no attack and no traffic spike.

I had given Claude Code a list of six tasks, and it started six agents in parallel. Each one dutifully installed dependencies, ran the type checker and the tests, and built the project. On the same machine that serves the site.

The arithmetic nobody did

Take round numbers close to mine. The server has 10 GB of RAM, and the live services use about 3. That leaves 7. One typecheck or build of a project this size takes 1.5 to 3 GB. Six at once need 9 to 18. That doesn't fit into 7 however you slice it.

Anyone could have added this up. Nobody did, not me and not the model.

Why the server didn't save itself

I expected Linux to kill the hungriest process and let the rest carry on. It didn't. The journal has no out-of-memory kills at all. What it does have: system services reporting they were running "under memory pressure", and Docker saying it could not send a kill signal to a container process. Then nothing until the reboot.

Swap did it. When RAM runs out, the kernel starts moving pages to disk and back, and that eats all the CPU time. On paper there is still memory, so the kernel sees no reason to kill anything. In practice the machine is busy only with itself.

The server didn't crash, it hung, and that's worse. A crashed service restarts on its own. A hung one waits for a human.

Why the agent did it

For agents, parallelism is a feature. Six independent tasks go to six workers, each with its own copy of the code in a separate git worktree, nobody in anyone's way. In the code, that held. Where they got in each other's way was memory, and nobody had told them about it.

A model doesn't know how much RAM the machine has or that a live site runs next to it, unless it checks or someone tells it. And "finish the job and make sure it builds" is usually a good habit. Run six times over on a small server, that habit took the server down.

What I changed

• No more than two heavy jobs at once. Heavy means a build, a typecheck of the whole project, or a Docker image build.
• Each agent checks only its own part. Types and tests for the package it changed, no full build.
• Full builds run one at a time, from the main session, not inside every agent.
• The rule lives in CLAUDE.md, the project notes Claude reads at the start of every session.

Since then it brings the rule up on its own when the task list gets long.

Doing the math for your machine

Take total memory, subtract what's already in use, keep a gigabyte in reserve, and divide by the peak of one job. With my round numbers that's (10 - 3 - 1) / 3, so two jobs.

You don't have to guess the peak. Measure it:

free -h
/usr/bin/time -v npm run build 2>&1 | grep 'Maximum resident'

The first command shows how much memory is free right now. The second runs the build and prints the peak of its largest process, in kilobytes. A build that forks parallel workers uses more than that in total, so round up.

Two safeguards for when the math lets you down. First, don't build on the production server at all if you have a separate machine or CI. Second, install something like earlyoom, which kills the hungriest process early instead of waiting for the system to drown in swap.

I put in the second one late, and I shouldn't have. Three weeks later the server hung again, this time with no agents involved: a nightly job that checks for dependency updates built a Docker image and started a second copy of a service next to the live ones. Same arithmetic, only nobody was around to do it.

This is what /etc/default/earlyoom looks like on my box now:

EARLYOOM_ARGS="-m 5 -s 100 --prefer ^(chrome|chromium|tsc|next-server)$ --avoid ^(postgres|dockerd|containerd|containerd-shim|systemd|sshd|journald)$"

-s 100 is the part that matters here. By default earlyoom acts only when both RAM and swap are nearly gone, and my server was already thrashing while swap was still free. With -s 100 it acts on low RAM alone. Builds and the browser go first, and the database and Docker are never touched.

Each of the six agents did the right thing. The only thing wrong was how many there were. I collect findings like this about Claude and AI on Promtime, the news site this server runs.