Requests timing out but CPU normal: thread pool starvation in ASP.NET

Requests timing out but CPU normal: thread pool starvation in ASP.NET

This post was originally published on MatrixTrak.com - the production reliability toolkit for trading bot operators and .NET engineers.

When requests time out but CPU is low and restarting fixes it temporarily: how thread pool starvation happens, how to prove queueing, and the smallest fixes that stop repeat incidents.

At 09:12, p95 doubles across the app. Error rate creeps up. CPU is calm. Memory is flat. The database looks "fine". By 09:31 someone recycles IIS (or restarts the service) and latency snaps back to normal. By 10:20 the slowdown returns.

That failure mode is thread pool starvation. Too many worker threads get captured waiting, so new request work queues up and everything times out.

On call, two questions decide what to do next:

• What evidence proves this is queueing behind blocked work?
• What is the smallest change that stops the repeats without a risky rewrite?

If you are the tech lead, the goal is a bounded stabilization plan you can ship this sprint. If you are the CTO, the goal is measurable risk reduction: fewer timeouts, less paging, and fewer "restart fixed it" incidents.

Rescuing an .NET service in production? Start at the .NET Production Rescue hub.

If you only do three things

• Prove queueing: chart p95/p99 + throughput together (not CPU alone).
• Add dependency timing logs with a timeout budget field (elapsedMs + timeoutMs + outcome).
• Remove sync waits in hot paths and cap concurrency around the slow dependency (bulkhead).

Why timeouts happen when CPU looks fine: the starvation mechanism

Thread pool starvation is not "the server is overloaded". It is "the server is waiting".

In ASP.NET, each request needs a worker thread to run its work. If enough worker threads get stuck waiting (sync-over-async waits, lock contention, slow dependency calls with no timeout budget), the process still has CPU available but it cannot schedule new request work quickly. Requests queue. Latency rises across endpoints. Timeouts appear in places that look unrelated.

Operationally, you have accidentally turned your service into a queue you did not design, size, or instrument.

Why CPU is low but everything times out: waiting vs computing

CPU measures compute. Starvation is usually waiting. A thread that is blocked on a lock or blocked on synchronous I/O is not burning CPU, but it is still consuming your most limited resource during peak traffic: runnable request capacity.

When enough threads are captured, new work queues. Queueing is what drives p95 and p99 through the roof.

That is also why "restart fixed it" is a recurring clue. A restart does not make the code faster. It discards the backlog of blocked work and resets state. If the mechanism that captures threads is still present, the backlog rebuilds under the same traffic shape.

How to recognize starvation: everything slows down together

Starvation has a specific smell: everything gets slower together. It is not one slow endpoint. It is the whole process losing the ability to schedule work. That is why symptoms spread across unrelated routes and unrelated downstreams.

Look for a cluster like this:

• Latency rises across many endpoints at the same time
• Throughput drops (req/sec decreases)
• CPU is not pegged (often 20 to 60 percent)
• Downstream timeouts show up in bursts (HTTP, SQL, cache)
• Recycling or restarting makes it look "fixed" for a while

For background workers, the shape is similar:

• Queue depth climbs
• Oldest message age climbs
• Workers are "running" but completions slow down

Fast triage table (what to check first)

Symptom	Likely cause	Confirm fast	First safe move
p95 rises across many endpoints while throughput drops and CPU is moderate	Queueing behind blocked work (thread capture)	Active requests / oldest request age climbs; traces/dumps show threads waiting	Lower/standardize timeouts, cap concurrency (bulkhead), remove sync waits in hot paths
CPU is high and stays high	Compute saturation	CPU profile shows hotspots; throughput drops because cores are pegged	Reduce work per request, cache, scale out (after confirming dependency isn't the bottleneck)
GC time spikes; allocations spike	Memory pressure / GC pauses	GC metrics show high pause %, Gen2/LOH churn	Reduce allocations, fix high-churn paths, validate with load
One endpoint (or one job type) is slow; others are fine	Localized hot path or single dependency	Per-route latency and logs isolate the path	Fix that path first; add explicit timeouts and bounded retries
Scaling out makes it worse	Shared downstream is the bottleneck (SQL/vendor API)	Downstream latency rises with instance count	Stop amplification: bulkheads + stop rules; reduce retries into timeouts

Why it happens (common causes)

Most teams do not create starvation intentionally. They create it one reasonable decision at a time. The job here is not to collect symptoms. The job is to find the mechanism that captures threads.

Sync-over-async (most common)

Somewhere, async work is forced into a blocking wait:

// Common offenders in request paths
var result = SomeAsyncCall().Result;
SomeAsyncCall().Wait();
SomeAsyncCall().GetAwaiter().GetResult();

In classic ASP.NET this can deadlock. In ASP.NET Core it often appears to work until load increases. Either way, it blocks a thread waiting for I/O. Enough blocked threads and your service becomes a queue.

The durable fix is not "add more async". The durable fix is making the hot request path async end-to-end and removing synchronous waits.

Long I/O with no timeouts

If a call can wait forever, eventually it will. Typical culprits:

• HTTP calls without a real timeout budget
• SQL calls stuck behind locks
• Vendor SDK calls that block internally

The consequence is not just one slow call. It is thread capture. That is how a dependency glitch becomes a platform incident.

Lock contention and critical sections

Starvation does not require I/O. If many threads are blocked on locks (or waiting on a single shared resource), the effect is similar: a backlog forms, but CPU looks "fine".

Retry storms amplify the backlog

Retries are concurrency multipliers. If a dependency slows down and you retry aggressively, you create more in-flight work. That creates more blocked threads. That increases queueing. That increases timeouts. This is how a small blip becomes a full incident.

Decision framework: confirm starvation, then find the captor

Starvation is a scheduling failure. The question is what is capturing threads. Use this quick framing to avoid the two most common mistakes: blaming the wrong dependency and "fixing" symptoms by adding capacity.

• If CPU is high and stays high, you have compute saturation. Starvation may exist too, but compute is the first-order problem.
• If GC is dominating and allocations spike, you have memory pressure. Starvation symptoms can be downstream of GC pauses.
• If CPU is moderate and p95 rises everywhere while throughput drops, assume queueing. Then prove what is holding threads.

Focus on the third case below.

How to diagnose: prove queueing, then find what's capturing threads

During an incident, starvation becomes a debate. One person blames SQL. Another blames networking. Another wants to scale out. Skip the debate. Collect evidence that explains the mechanism: what is queued, what is waiting, and what is holding threads.

1) Prove queueing, not compute

You want a picture that says: latency is rising, throughput is falling, CPU is not pegged.

Signals that count:

• p95 and p99 rise across many endpoints at the same time
• req/sec drops during the slowdown window
• active requests climb and stay elevated

Interpretation: when latency rises and throughput drops while CPU stays moderate, you are looking at queueing behind a shared bottleneck.

2) Identify the dependency budget that is being violated

Queueing alone does not tell you where to fix. Add structured dependency call logs and watch for one pattern: a dependency consistently exceeds its timeout budget during the slowdown window.

This log shape is enough to be actionable:

{
  "ts": "2026-01-21T09:18:34.120Z",
  "level": "warning",
  "correlationId": "c-8f5e9b8f5fdd4e29",
  "endpoint": "GET /orders/{id}",
  "dependency": "sql:OrdersDb",
  "elapsedMs": 4120,
  "timeoutMs": 3000,
  "attempt": 1,
  "outcome": "timeout",
  "note": "suspect queueing / blocked threads"
}

Interpretation: if one dependency starts exceeding its budget, that is your likely captor. The fix is not more threads. The fix is timeouts, bounded concurrency, and stop rules.

3) Capture a short artifact while it is slow

One short artifact taken during the slowdown prevents days of guesswork. Aim for a 20 to 60 second capture.

Good artifacts:

• PerfView trace
• dotnet-trace
• A process dump (dotnet-dump) for later analysis

The artifact should answer one question: what are thread pool threads waiting on?

4) Find the captor: sync waits, locks, or unbounded fan-out

Now look for the smoking gun mechanism:

• Sync waits on HTTP calls
• Sync waits on database calls
• Contention on one shared lock
• Unbounded parallel fan-out around a slow dependency

At this point you should be able to name the captor in one sentence.

Containment moves (during the fire)

Containment is not a perfect fix. Containment is stopping the system from getting worse.

The common chain looks like this: a dependency slows down, threads get captured, queues grow, retries amplify, then everything collapses. Containment breaks the chain.

Moves that usually buy you time:

• Reduce concurrency around the hot dependency (bulkhead)
• Lower timeouts so blocked work is released
• Disable expensive or non-essential features behind a flag
• Cap retries hard, or temporarily disable retries for the failing dependency

Scaling out can help if the service is truly capacity bound. It can also make the incident worse by increasing pressure on the same slow dependency. Treat scale-out as a hypothesis, not a reflex.

Fixes that stop repeats (smallest first)

Once you have seen starvation once, the goal is to remove thread capture, not to tune around it. Thread pool tweaks and more servers can hide symptoms. They do not change the fact that a captured thread is still a captured thread. The same mechanism will return under load.

1) Remove sync waits in request paths

Find offenders like .Result, .Wait(), and .GetAwaiter().GetResult() in hot paths and remove them. This is usually the highest ROI stabilization work because it reduces thread capture immediately without changing business behavior.

2) Make timeouts explicit, consistent, and logged

Timeouts are how you prevent infinite waits from capturing threads. A timeout is also a decision point: fail fast, degrade, or stop. Prefer a consistent budget per dependency and log when the budget is exceeded. If a dependency can stall your process indefinitely, it will.

3) Cap concurrency at the call site (bulkheads)

If your SQL server can safely handle 50 heavy queries, do not allow 500. Put the limit where it matters: the place that fans out requests.

One safe pattern is a dependency bulkhead:

private static readonly SemaphoreSlim OrdersDbBulkhead = new(50);

public async Task<OrdersResponse> GetOrdersAsync(OrdersRequest request)
{
    await OrdersDbBulkhead.WaitAsync();
    try
    {
        // actual call
    }
    finally
    {
        OrdersDbBulkhead.Release();
    }
}

Resources

Package details and download live on the resource page. The external links are the official tooling references.

• Thread pool starvation triage package
• .NET Production Rescue hub
• Axiom (Coming Soon)
• Retries making outages worse: when resilience policies multiply failures in .NET - retry storms amplify starvation
• Timeouts first: why infinite waits create recurring outages in .NET - infinite waits capture threads
• Cannot trace requests across services: why correlation IDs die at boundaries in .NET - trace blocked request chains
• Polly retries making outages worse: how retry storms multiply failures in .NET - bounded retries prevent capture

External references:

• dotnet-trace
• dotnet-dump
• PerfView

Edge cases and misconceptions that show up in real incidents. Axiom is where we ship operator-grade assets for production teams: runbooks, templates, and decision trees that are designed for legacy constraints. Join to get notified when new incident packages ship.

Checklist (copy/paste)

• [ ] Latency + throughput charted together (p95/p99 and req/sec).
• [ ] Backlog signal exists (active requests, oldest request age, queue depth).
• [ ] Dependency logs include: correlationId, endpoint, dependency, elapsedMs, timeoutMs, attempt, outcome.
• [ ] At least one hot path is async end-to-end (no .Result, .Wait(), .GetAwaiter().GetResult()).
• [ ] Explicit timeouts exist for HTTP, SQL, and jobs (no infinite waits).
• [ ] Concurrency is capped around the slow dependency (bulkhead), not via global locks.
• [ ] Retries are bounded and conditional (stop retrying timeouts into a slow dependency).
• [ ] One artifact can be captured during a slowdown (PerfView / dotnet-trace / dump) and attached to the incident ticket.
• [ ] After the fix, a load test proves: throughput recovers and p95 stays stable without recycling.

If you only change a few things, change these. Thread pool starvation is usually thread capture (sync waits, long I/O, locks), not high CPU. A restart is evidence, not a fix. It discards blocked work. Durable fixes remove thread capture: async end-to-end, explicit timeouts, bounded retries, and capped concurrency.

If you are dealing with this now, start at the .NET Production Rescue hub or contact me with one slow-request log (include a correlation ID and dependency timing).

Free Tools for Trading Bot Reliability

Every article on MatrixTrak is backed by free, open-source tools you can use right now:

• Incident Runbook Builder - decision trees + escalation paths → Try it free

Originally published at matrixtrak.com.