I Built a Concurrent Resource Scheduler in Go Using Sharded Priority Heaps

Support on GitHub: github.com/phero20/concurrent-resource-scheduler (Give it a star if you find it useful!) View Docs: pkg.go.dev/github.com/phero20/concurrent-resource-scheduler What happens when thousands of concurrent requests compete for a small pool of reusable resources? You can put a mutex around a slice and hope for the best. Or you can design the scheduler around concurrency from the beginning. I chose the second option. I built Concurrent Resource Scheduler (CRS), a domain-agnostic Go library for selecting, prioritizing, routing, and maintaining reusable resources under heavy concurrent load. The core idea is simple: MANY CONCURRENT REQUESTS │ ▼ ┌───────────────────┐ │ Resource Scheduler│ └─────────┬─────────┘ │ ┌──────────────┼──────────────┐ │ │ │ ▼ ▼ ▼ Priority Acquire State Heap Strategy Management │ │ │ └──────────────┼──────────────┘ │ ▼ BEST AVAILABLE RESOURCE But making that work correctly under concurrency is where things get interesting. CRS is designed for use cases such as: - LLM/API gateways - API key pools - proxy rotation - database replicas - GPU workers - backend pools - worker resources - connection pools - rate-limited providers - reusable compute resources The scheduler itself does not know what a resource means. It only knows: "I have resources. I need to safely maintain them, prioritize them, and return an appropriate one to a concurrent caller." Table of Contents - The Problem - The Naive Approach - Why a Global Mutex Becomes a Problem - The Core Idea Behind CRS - Architecture at a Glance - Sharded Priority Heaps - Why Sharding Helps - The O(1) Lookup Map - Priority and Acquire Are Different Problems - Acquire Strategies - Round Robin - Weighted Acquire - Adaptive Acquire - Affinity Routing - Shared vs Exclusive Acquisition - Resource Lifecycle - Atomic State Transitions - The Inactive Store - Batch Operations - Updates Without Destroying Heap Ordering - Cooldowns - Asynchronous Events - Observability - Prometheus Integration - Concurrency Model - Complexity - Testing the Library - Race Detector Validation - Real-World Load Testing - 10,000 Concurrent Workers - Burst Testing - Failure Testing - Cooldown Stress Testing - What the Load Tests Actually Tell Us - A Minimal Example - LLM Gateway Example - Why CRS Is Domain-Agnostic - Project Structure - Design Principles - Lessons Learned - When You Should NOT Use CRS - Future Directions - Final Thoughts The Problem Let's start with a realistic scenario. Imagine an LLM gateway with 100 API keys. Each key may have different: - rate-limit availability - priority - health - cooldown state - provider - capacity - temporary availability Thousands of requests arrive concurrently. A simplified system looks like: ┌──────────────┐ Request 1 ────────► │ Request 2 ────────► │ Request 3 ────────► Gateway │ Request 4 ────────► │ Request 5 ────────► │ ... │ │ Request N ────────► │ └──────┬───────┘ │ ▼ ┌───────────────┐ │ Resource Pool │ └───────┬───────┘ │ ┌──────────────┼──────────────┐ ▼ ▼ ▼ API Key 1 API Key 2 API Key N The scheduler now has to answer: - Which resource should this request use? - Which resource has the best priority? - Is the resource currently active? - Can multiple requests use it simultaneously? - Should this resource temporarily leave the pool? - Which shard should we search? - Should requests stick to the same shard? - What happens when the resource is released? - How do we update its priority? - How do we observe all of this without slowing down the hot path? That is the problem CRS tries to solve. The Naive Approach The easiest implementation looks something like: type Scheduler struct { mu sync.Mutex resources []*Resource } Then: func (s *Scheduler) Acquire() *Resource { s.mu.Lock() defer s.mu.Unlock() // Scan resources. // Find the best one. // Return it. return best } At first glance, this looks perfectly reasonable. For 10 resources and 2 goroutines, it probably is. But imagine: Resources: 10,000 Concurrent requests: 5,000 Now every operation fights over one lock. GLOBAL MUTEX │ ┌─────────────┼─────────────┐ │ │ │ ▼ ▼ ▼ Worker 1 Worker 2 Worker 3 │ │ │ └─────────────┼─────────────┘ │ WAITING The scheduler becomes serialized around the lock. Why a Global Mutex Becomes a Problem There are several problems. 1. Lock contention Only one goroutine can manipulate the pool at a time. 2. Linear scanning If resources are stored in an array, finding the best resource can become: O(N) per acquisition. 3. Priority maintenance If resources have priorities that change, the scheduler has to continuously maintain ordering. 4. State transitions Resources can move between: ACTIVE INACTIVE REMOVED and those transitions must be synchronized. 5. Observability Metrics and event callbacks should not block the scheduler. The challenge is therefore not just: "How do I build a priority queue?" It is: "How do I build a concurrent priority resource manager where priority, acquire, lifecycle, and observability coexist?" The Core Idea Behind CRS The central architectural decision was: Don't put one global lock around the entire priority structure. Instead, CRS partitions resources into independently locked shards. Conceptually: CRS │ ┌──────────┼──────────┐ │ │ │ ▼ ▼ ▼ Shard 1 Shard 2 Shard N │ │ │ Heap Heap Heap │ │ │ Mutex Mutex Mutex Each shard owns its own heap and its own lock. This is the heart of CRS. Architecture at a Glance APPLICATION │ Add / Acquire / Release / Update │ ▼ ┌────────────────────┐ │ CRS Scheduler │ └─────────┬──────────┘ │ ┌───────────────┼────────────────┐ │ │ │ ▼ ▼ ▼ Acquire Lookup Inactive Strategy Map Store │ │ │ ▼ ▼ │ Candidate Shard O(1) Node │ │ │ ▼ │ ┌─────────────────────────────────┐ │ │ ACTIVE HEAP SHARDS │ │ │ │ │ │ Heap 1 Heap 2 Heap N │ │ │ +Mutex +Mutex +Mutex │ │ └─────────────────────────────────┘ │ │ │ └──────────────┬─────────────────┘ │ ▼ EVENT DISPATCHER │ ┌─────────┴──────────┐ ▼ ▼ Telemetry Cooldown │ │ ▼ ▼ Prometheus Resource State This separation is intentional. Sharded Priority Heaps Each shard maintains a priority heap. For example: Shard 1 [Priority 10] / \ [Priority 20] [Priority 30] / \ [40] [50] Another shard: Shard 2 [Priority 5] / \ [Priority 15] [Priority 25] Every shard has its own synchronization boundary. Shard 1 Shard 2 Shard 3 ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ Mutex │ │ Mutex │ │ Mutex │ ├─────────────┤ ├─────────────┤ ├─────────────┤ │ Priority │ │ Priority │ │ Priority │ │ Heap │ │ Heap │ │ Heap │ └─────────────┘ └─────────────┘ └─────────────┘ There is no global heap mutex. Why Sharding Helps Suppose we have 32 shards. Instead of: 1 global lock we have: 32 independently locked heaps Different goroutines can operate on different shards simultaneously. Goroutine A ─────► Shard 1 ─────► lock Goroutine B ─────► Shard 7 ─────► lock Goroutine C ─────► Shard 19 ────► lock Goroutine D ─────► Shard 27 ────► lock The locks are independent. This doesn't magically eliminate contention. If every request targets the same shard, that shard can still become contended. That's why CRS also separates: acquire strategy from priority ordering This distinction is extremely important. The O(1) Lookup Map A heap is excellent at answering: "What is the best resource?" But a heap is not ideal for answering: "Where is resource X?" Searching a heap can require scanning. CRS therefore maintains an additional lookup structure. Conceptually: ID │ ▼ ┌──────────────────────┐ │ Lookup Map │ │ │ │ "backend-01" ───────► Node │ "backend-02" ───────► Node │ "backend-03" ───────► Node └──────────────────────┘ The lookup map is protected independently with a read/write mutex. This gives the scheduler an O(1)-style membership/location lookup by application-defined key. That is particularly useful for: Get Update Remove Release Exclude Include without scanning every heap. Priority and Acquire Are Different Problems This is one of the most important design ideas in CRS. A resource can have: Priority = 10 but that doesn't necessarily tell us: Which shard should we inspect first? These are separate decisions. CRS therefore separates: REQUEST │ ▼ ACQUIRE STRATEGY │ ▼ SHARD SELECTION │ ▼ PRIORITY HEAP │ ▼ BEST RESOURCE This allows different routing strategies to be plugged into the scheduler without changing the underlying heap implementation. Acquire Strategies CRS provides several acquire approaches: - Round Robin - Weighted - Adaptive - Consistent Hashing for affinity routing Each solves a different problem. Round Robin Round Robin is the simplest. Request 1 → Shard 1 Request 2 → Shard 2 Request 3 → Shard 3 Request 4 → Shard 4 Request 5 → Shard 1 ... It is simple and predictable. Use it when: - shards are roughly equivalent - you want even distribution - resource capacity is similar Weighted Acquire Not every shard is necessarily equal. Imagine: GPU 1 → 24 GB VRAM GPU 2 → 24 GB VRAM GPU 3 → 80 GB VRAM GPU 4 → 80 GB VRAM You may want larger resources to receive more work. Weighted acquire lets you express relative capacity. Conceptually: Shard 1: weight 1 Shard 2: weight 1 Shard 3: weight 4 Shard 4: weight 4 Traffic can then be distributed proportionally. This is useful for: - heterogeneous GPUs - backend instances with different capacity - API providers with different quotas - worker pools with different performance characteristics Adaptive Acquire Round Robin doesn't know anything about current load. Adaptive acquire attempts to account for shard activity. Conceptually: REQUEST │ ▼ ┌──────────────────┐ │ Inspect shard │ │ load information │ └────────┬─────────┘ │ ┌────────┼────────┐ ▼ ▼ ▼ Shard A Shard B Shard C busy low busy │ ▼ choose B The scheduler uses lightweight shard-level state to favor less-contended shards without introducing another global lock. This is useful when the resource pool is dynamic and simple round-robin distribution isn't enough. Affinity Routing Sometimes you don't want random distribution. You want: user-123 → same shard user-456 → same shard CRS supports affinity routing through c