The Twenty-Hour Fuse

Two nights of debugging, one incident left to write about. The night of August 14th, a Claude Code session hung for an hour inside the VS Code extension, and I traced it to a per-token streaming stall - the previous two pieces, Two Clocks, Neither Lying and Coalescing the Stream, cover that hunt and the middleware that fixed it. The morning after, the same four-core box nearly went down under a load average of 38.7, and it turned out to be the same incident, twenty hours downstream. Nothing exploded that morning. A fuse had been burning since the previous afternoon, and 11:09 was simply when it reached the powder. Reading the machine before touching it The load numbers spoke first: 11.08, 38.69, 23.59 - one-minute, five-minute, fifteen-minute. A crisis still climbing shows the opposite order, disaster in the one-minute figure first; mine were falling, so the peak had already passed while I was still reading. Then /proc/pressure - PSI, the kernel's stall accounting for CPU, memory, and IO, and the closest thing a Linux box has to a memory of the last five minutes: memory full at 10 percent (processes genuinely stopped, waiting on page reclaim) and IO some at 45 percent. Memory pressure had turned into an IO storm. The connection to the headline number is D state. Processes parked in uninterruptible IO waits - runc and a node healthcheck were caught in the act - count into load average exactly as if they were running. Four cores at 38.7 wasn't thirty-eight cores of computation; it was modest real work plus a crowd of processes waiting in the kernel's lobby. Elimination was fast because the box was eloquent. No OOM kills in the kernel log - the kernel hadn't started shooting anyone. docker stats : every container under 500 MB, single-digit CPU - not the compose stacks. Disk at 65 percent - not full. A process listing sorted by memory, then grouped: llama.cpp at 3.9 GB, seven Java services at 2.3 GB, k3s at 1.1 GB - steady state, all normal, none of it news. And fifty-nine vscode-server processes holding 7.3 GB, which is not steady-state anything. One server, two windows, and the rule that fell out of the map The decisive hour went into mapping the process tree, PPID by PPID, because "59 processes" is a number, not a structure. The structure: Remote-SSH runs one server per client version on the box, parented to init - detached from any SSH session, and therefore under no session's death sentence - and that one server hosts every window the client opens, one extension host per window. A server started on August 9th was hosting two. One was the window I was typing in. The other was an old window from August 14th whose extension host had grown seventeen processes and 3.2 GB - containing the very Claude session that had hung the night before, plus four more claude processes born of retries between 2:04 and 2:11 a.m., every retry a new process while the old ones never exited. A second, separate server tree - thirteen processes, another device's window from 8:26 that morning - completed the census. The operational rule fell straight out of the map: I could not kill a server, only windows. Kill the shared server and I'd be sawing the branch I sat on. The kill list was the old window's extension-host tree, the other device's whole server tree, and nothing else. Twenty hours, then eleven minutes As a timeline, the incident runs at two speeds. Accumulation: from 15:17 on the 14th, when the old window connected and the stalled session began its hour-long hang, to 8:26 the next morning. Retries stacked unexiting processes; detached servers sat immune; and six versions of the server binary piled up in ~/.vscode-server - each client upgrade downloads a new one - 3.8 GB of them. Memory crept past 85 percent slowly enough that nothing intervened: no swap to absorb the creep, and memory alerts set at 80 and 90 percent are spike detectors - for growth this gradual they arrive with hours of momentum already behind them. Ignition: 10:59 to 11:09. A new connection, a reconnect, two fresh extension hosts spinning up their language servers - Pylance and friends, CPU- and IO-heavy - on a box whose page cache was already being squeezed to stay alive. Eleven minutes of simultaneous startup on a machine with no headroom: the kernel began reclaiming the pages everything was reading, IO seized, processes dropped into D state, and the load average registered the detonation. The five-minute peak of 38.7 covers exactly those eleven minutes. Four layers, four failures The useful postmortem question isn't what broke; it's what should have caught this and didn't. Four layers, each with a job, each failing for a specific reason. The trigger layer is the stall itself, and it's the one layer that held - the middleware went in the night before. This incident arrived with its ignition source already removed; the fuse just had to finish burning. The accumulation layer failed architecturally. Servers that outlive their sessions because nothing ever kills them; an old client version that doesn't start its servers with the auto-shutdown flag the new ones carry; and - the part that's mine - a replication engine made of my own reasonable responses. Every "reopen the window and retry" was the correct move as far as I knew, and every one cloned the problem: a fresh process next to the ones that never left. The buffer layer failed by absence. Without swap, the kernel's relief valve under memory pressure is reclaiming the page cache - the exact thing every running process is trying to read - which is how a memory shortage converts into an IO storm. Swap is the difference between slow and seized. The capacity layer failed by design. Steady state on this box is about 15 GB of its 23 - k3s, llama.cpp, ten compose stacks, seven JVMs - leaving thin elastic headroom for the one workload that actually bursts: interactive sessions. The incident priced that layout. The surgery, and two humbling footnotes The cleanup went carefully, in this order: a keep-list first - the entire process chain of the window I was typing in, the shared pty host, the agent host. Then a recursive walk collecting every descendant of each root marked for death, with each root's identity re-verified against its command line, because a PID is a label that gets reused, not a name. TERM first; KILL for the survivors four seconds later. And one rule I never skip: the kill list gets checked for intersection with my own process chain, because the checker is also a process on this box, and the one tree it must never reap is its own. Fifty-nine VS Code processes became nineteen; fourteen once the dust settled. Then 4 GB of swap - fallocate, mkswap, swapon, fstab - after which available memory jumped from 2.8 GB to 8.2 GB, mostly because the kernel stopped strangling the page cache. Then the version pile: five of the six server directories deleted, 3.2 GB back. Footnote one: before deleting those directories I verified nothing referenced them, by grepping ps output for the directory names. Every directory came back "in use" - because ps prints my own checking command's command line, names included. The check had matched its own author. The corrected version then carried its own bug, an invalid ps invocation that silently verified nothing, and the deletion went through unguarded; a full manual census afterward confirmed nothing innocent had died. The lesson generalizes: any check that greps a process list and then acts on the result must exclude the checker - the observer is always inside the observation. Footnote two: at 11:28 the killed device reconnected by itself - the desktop client, still open, reaching out and re-downloading 600 MB of server binary because I'd just deleted it - and the old window re-spawned an extension host onto the shared server. Killed again; four quiet minutes after that. You cannot win this fight on the server alone. Server-side cleanup plus client windows actually closed, or the tree regrows. | 11:09 | 11:33 | | |---|---|---| | Load average (1 min) | 11.1 (5-min peak 38.7) | 0.68 | | Memory used / available | 20 Gi / 2.8 G | 15 Gi / 8.2 G | | Swap | none | 4 GB, in fstab | | IO PSI (10 s / 5 min) | 21% / 45% | 0.01% / 2.1% | | VS Code processes | 59 | 14 | | Server versions on disk | 6, ~3.8 GB | 1, 570 MB | Checked again as I write this: swap present and quiet, one server directory, load under one. What I actually fixed The honest accounting is that the morning's cleanup treated the wreckage, not the disease. The actual fix had happened the night before, in a middleware file: with the per-token stream gone from the pipeline, sessions end when they end, processes exit, and the accumulation layer lost its fuel. What this incident added was the rest of the defense - swap as the shock absorber, and seven new alert rules, across four signals: PSI (io and memory pressure), the count of processes stuck in D state, swap presence and consumption, and a predict_linear forecast on the memory trend. All four read metrics node_exporter was already emitting, so none of it needed new infrastructure. The instinct I reached for first - alert on the vscode-server process count directly - would have been the more legible signal, but shipping it meant a host cron job, a textfile collector, and a node-exporter mount change for one app-specific number; it's shelved for now in favor of generic pressure signals that were already flowing. And a habit: close the window, don't reopen it. Every alarm I had was built for the eleven minutes. The trend forecast is the one now aimed at the twenty hours before it. Top comments (0)