Building Sluice: QoS-Aware Capacity Governance for Self-Hosted LLM Inference

📦 Project: https://github.com/VampiricCyborg/sluice 1. The Problem: When Capacity Becomes the Bottleneck A self-hosted vLLM deployment runs on a GPU pool of fixed size. That's the fact that changes everything about how you have to think about load. Once that pool's KV-cache capacity comes under pressure, requests don't naturally understand business tiers. They don't know that one tenant is an enterprise customer with a contract, another is an internal team, and a third is a batch job that can wait. Without something external saying otherwise, the system's behavior under contention is driven by arrival order and backend scheduling - and every tenant, regardless of what they were promised, experiences the same degraded latency, queueing, or failure. There's no built-in mechanism that says: Preserve Guaranteed traffic, reduce the work done by Standard traffic, and shed Best-Effort traffic first. Sluice exists to make that decision - before the request ever reaches vLLM. Concretely, it evaluates three live signals per decision cycle: - pressure - vLLM GPU KV-cache usage, read from Prometheus. - queue_depth - vLLM's waiting-request depth, also from Prometheus. - sla_violation_rate - computed from completed requests in the PostgreSQL decision ledger. Tenants are mapped to tiers in config/tenants.yaml : tenants: support-bot: Guaranteed coding-assistant: Standard batch-analytics: Best-Effort and each tier has an explicit SLA target, hard-coded in sluice_proxy/app.py : sla_targets = { "Guaranteed": 2000, "Standard": 1000, "Best-Effort": 500, } The important thing to understand about Sluice's scope from the start: it is not trying to make vLLM's own execution more efficient. It governs which requests reach the inference engine, how much work they're allowed to request, and which backend pool they land on. Everything downstream of that decision is still vLLM's job. 2. Why Existing Layers Don't Solve It It's worth being precise about why this problem doesn't already have a home in the stack, because Sluice's design only makes sense in contrast to the layers around it. Kubernetes can place workloads, restart failed replicas, and add capacity if an autoscaler is configured. None of that helps when GPU capacity is fixed or slow to provision - which, for self-hosted inference, is closer to the default case than the exception. Kubernetes has no concept of a "tenant" or a "tier" at the request level; it schedules pods, not chat completions. vLLM's own scheduler is the right layer for token-level execution, batching, and KV-cache management inside a worker - and it's better at that than anything sitting in front of it could be. Sluice deliberately stays outside that loop. It has no interest in improving token scheduling; it adds tenant identity, tier semantics, and an auditable reason for each request-level action, then gets out of the way. A generic API gateway (Envoy, Kong) gives you authentication, routing, retries, and rate limiting - genuinely useful, and Sluice doesn't try to replace any of it. But rate limits alone can't express "degrade this tier's max_tokens before rejecting that tier's requests, based on live GPU pressure." That's a specialized policy signal and action set that a generic gateway has no vocabulary for. The actual boundary in code is narrow. The original Phase 1 implementation of sluice_proxy/app.py did five things: identify the tenant from request headers, look up their tier, read cache pressure, apply a policy decision, and optionally rewrite max_tokens before forwarding. It never touched token scheduling, batching, or KV-cache allocation - those stayed vLLM's problem, from the very first commit. The is_shapeable() function makes this boundary explicit today. Only POST , PUT , PATCH requests to /v1/chat/completions or /v1/completions are eligible for shaping at all. Everything else passes through untouched. And even for shapeable requests, Sluice only ever changes two fields: max_tokens (via apply_max_tokens() ) and model (via apply_model() , for fallback routing). That's the entire surface area of what Sluice is willing to touch in a request body. The same discipline shows up in cluster routing. ClusterRoutingStage in sluice_proxy/policy.py carries this as an explicit design intent in its own docstring: QoS route between pools, never utilization/load balance. Routing is a service-policy decision - Guaranteed always gets on-demand, Standard gets on-demand, Best-Effort prefers spot unless spot pressure or a health failure forces eviction - not a round-robin balancing act. And the Kubernetes deployment in k8s/sluice-ops.yaml is intentionally just a Deployment, a Service, a ConfigMap, and probes. No operator, no autoscaler, no scheduler extension. 3. The Constraint: What Sluice Is Actually Responsible For Before getting into design, it's worth stating plainly what Sluice owns, because everything downstream follows from this boundary. The clearest way to see it is through its actual interfaces. The request path is a single catch-all: /v1/{path:path} For every request that hits it, Sluice does exactly seven things: resolve tenant and tier, read current telemetry, evaluate policy, apply shaping/queueing/fallback/rejection/routing, forward to the selected backend, record the result in PostgreSQL, and emit a structured log. Operational visibility is a separate, small surface: /healthz , /livez , /readyz , /metrics , /status , /decisions/recent . And for local development, a narrow set of simulation endpoints - PUT /DELETE on /admin/pressure , /admin/queue-depth , /admin/sla-violation-rate - let you force a signal value without waiting on real telemetry. These are intentionally minimal development controls, not a control-plane API, and they stay unauthenticated even when API-key auth is otherwise enabled on /v1/* - a decision that's convenient for local testing and a real risk if ever exposed (more on that in Section 11). The external dependencies are equally explicit: Prometheus for live inference telemetry, PostgreSQL for durable decisions and SLA history, Redis for distributed queue/admission state, and any number of vLLM-compatible HTTP backends. Sluice doesn't implement any of these things - it coordinates between them. Architecture REQUEST │ ▼ ┌──────────────┐ │ Sluice │ └──────┬───────┘ │ ┌────────────▼────────────┐ │ Pressure Evaluation │ │ KV Cache │ Queue │ SLA │ └────────────┬────────────┘ │ ▼ ┌───────────────┐ │ Tenant Policy │ └───────┬───────┘ │ ▼ ┌───────────────┐ │ Action Select │ │ Pass/Degrade │ │ Queue/Fallback│ │ Reject │ └───────┬───────┘ │ ▼ ┌───────────────┐ │ Cluster Route │ └───────┬───────┘ │ ┌────────┴────────┐ ▼ ▼ On-Demand GPU Spot GPU 4. Design: From Pressure to Routing Sluice's decision logic didn't start as a pipeline. It started as one function. In the first commit of the policy engine (c101afb ), CapacityPolicy had a single decide() method with nested tier conditionals: if tier == "Best-Effort": if pressure >= 60: reject elif tier == "Standard": if pressure >= 90: reject if pressure >= 70: degrade elif tier == "Guaranteed": if pressure >= 100: reject if pressure >= 95: degrade That was fine when there was exactly one signal. It stopped being fine the moment a second one showed up. Commit 91c00fa extended decide() to accept pressure , queue_depth , and sla_violation_rate together, and the shape it took was still recognizably one function - checking whether any signal crossed a reject threshold, then whether any crossed a degrade threshold. It worked, but the conditional surface was growing in a direction that wasn't going to scale to fallback routing, queueing, or multi-cluster decisions without becoming unreadable. Commit ca80787 is where the real refactor happened: four explicit stages, each owning one concern. PressureEvaluationStage.evaluate(tier, signals) -> PressureEvaluation TenantPolicyStage.apply(evaluation, *, shapeable, fallback_active, fallback_model) -> PressureEvaluation ActionSelectionStage.select(evaluation, *, shapeable, fallback_active, fallback_model) -> PolicyDecision ClusterRoutingStage.route(tier, decision, clusters, health) -> PolicyDecision orchestrated by a single PolicyPipeline.run() . Pressure Evaluation turns raw signals into a named state (Normal, Elevated, Critical). Tenant Policy is the extension point for tier-specific semantics. Action Selection picks the actual intervention. Cluster Routing decides which pool handles it. I want to be honest about one detail here rather than present this as a clean four-stage design from the start: TenantPolicyStage.apply() is currently a no-op - it returns the evaluation unchanged. The refactor established the stage boundary, but not every piece of tier-specific behavior actually lives there yet. Some of it is still inside CapacityPolicy._select_action() and _thresholds_for() . That's a real, current gap between the architecture's intent and its full implementation - not a mistake, just unfinished separation. The pipeline shape was worth building before every piece of logic was moved into it, because it made everything that came after (fallback, queueing, multi-cluster routing) addable without another rewrite. 5. Making Pressure Stable: State, Hysteresis, and Cooldowns Three named states drive everything downstream: PressureState.NORMAL PressureState.ELEVATED PressureState.CRITICAL The combination rule across three signals is deliberately conservative - a max-severity OR, not an average: any available signal crossing a reject threshold produces CRITICAL ; failing that, any signal crossing a degrade threshold produces ELEVATED ; otherwise NORMAL . If every signal is unavailable, the policy fails open rather than guessing. The stabilization defaults, set in CapacityPolicy.init() : SLUICE_STATE_CONSECUTIVE_SAMPLES = 2 SLUICE_STATE_COOLDOWN_SECONDS = 2 SLUICE_STATE_HYSTERESIS = 5 I'll say up front that these are operational defaults, not values fitted to production traces - there's no evidence in the repository