Measuring the real concurrency ceiling of an LLM agent runner

I wanted to raise the concurrency limits on my local AI agent runner. The UI now supports multiple terminal panes running in flight, and my gut told me the runner process itself was becoming the bottleneck. Before touching a single config setting, I wrote a benchmark to test that assumption: When N sessions run at once, what actually breaks - the model server, the hardware, or the scheduling policy? It turns out my gut was entirely wrong.

The Benchmark Setup

I wrote a bench script that fires N concurrent chat sessions against the runner. Each turn hits a local model (hermes3 via Ollama) with a fixed prompt, running through the full pipeline: pre-turn intent classification, chat response, and post-reply memory extraction. I tracked four specific metrics:

• ttfa (Time to First Activity): POST request -> first internal event emitted. Measures the runner's event loop before touching Ollama.
• first-Ollama: POST request -> first request hitting the model queue.
• done: POST request -> user receives the answer.
• wall (drained): Total time until the post-reply memory extraction tail completely finishes background work in Ollama.

The Results: A Hard Wall at N=1

At N=4, the wall-clock time scaled to 4.80× the single-session baseline. Because 4.8× exceeds 4×, concurrent requests actually performed worse than a pure, perfectly ordered serial queue.

The bottleneck breakdown was immediately clear:

• It's not the runner: ttfa stayed at 0.0s across all runs. The runner's event loop processed and routed incoming POST requests in milliseconds without queuing.
• It's not hardware: CPU peaked at 63% and free RAM never dropped below ~5.9 GB.
• It's Ollama: Latency stacked inside Ollama's internal request queue for the single loaded model (hermes3 8.0B Q4_0).

Furthermore, wall (drained) proved that background memory extraction keeps Ollama saturated long after the user gets their answer - a hidden tax naive RPS benchmarks completely miss.

The Twist: Job Queues Fail by Policy, Not Load

Chat is only half the system. The real heavy lifting happens in coding jobs, which talk to the Claude API instead of Ollama. When I checked why coding jobs weren't running concurrently, I didn't need a load test. I just needed to look at the scheduler code:

const MAX_CONCURRENT = Number(process.env.MAX_CONCURRENT_JOBS ?? 2)

// In the scheduler pump:
if (repoHasRunningJob(job.repo)) continue; // One job per repo, unconditionally

Almost every ticket in my queue targets the same primary repository. Regardless of what MAX_CONCURRENT was set to, the system was hardcoded to run one job per repo at a time. The rule exists for a valid reason: two AI agents branching off the same moving HEAD create messy merge conflicts. But serializing the entire repository by default was a blunt policy constraint, not a system capability limit.

Smart Parallelism: A Disjoint Path Rule

To fix job concurrency without causing merge chaos, I replaced the blind "one job per repo" check with a strict path-overlap predicate. Two queued jobs (A and B) in the same repo can now run concurrently only if all three conditions are met:

• Disjoint Named Paths: Both tickets explicitly declare target files/directories, and their path sets share no common files, ancestors, or subdirectories. (If a ticket names no path, it is treated as touching everything and stays serial.)
• No Dependency Edges: Neither ticket references the other as a hard or soft dependency.
• Historical Touched Files: Reopened tickets use their previous run's actual git diff file list rather than their written prose description.

If there's any ambiguity, the scheduler defaults to serial execution. A false positive (running serially when safe) costs a few minutes of queue time; a false negative (running concurrently and corrupting state) costs an hour of untangling merge conflicts by hand.

What I Actually Learned

• Chat concurrency binds on the local model server - raising limits requires better GPU hardware or multi-model inference instances, not TypeScript optimizations.
• Coding job concurrency binds on scheduling policies - fixable with smarter dependency tracking and file-path isolation.
• The runner infrastructure was completely fine all along.

Measure first. The limit you think you're hitting is rarely the one actually holding you back.

I'm Andréas - full-stack dev, CTO at a B2B SaaS, building my own agent tooling. Portfolio: https://andreas-bodin.vercel.app