LLMRix Model Router - an open-source multi-model routing and orchestration framework for Java.

Stop Hardcoding Model Names in Business Code: LLMRix Router Adds a Multi-Model Routing Layer for Java AI Apps When AI applications move from demo to production, the trouble usually isn't with prompts-it's with the model calls themselves. Which model should handle this request? What happens when the primary model is rate-limited? How do you control costs? Can you switch providers mid-stream when a streaming response drops? And when running multiple instances, how do you keep quota and health state consistent? LLMRix Model Router packages all these concerns into an open-source routing runtime for Java. Over the past couple of years, the barrier to integrating AI models has dropped significantly. With a single SDK and a few lines of code, an app can call OpenAI, DeepSeek, or any other compatible interface. But once you enter production, a different set of problems emerges: upstream rate limits, timeouts, regional outages, and models that differ widely in capabilities, pricing, context windows, and response speed. When business code is directly coupled to a specific provider, swapping models means rewriting interfaces, exception handling, monitoring, and configuration all at once. Simple round-robin or reverse proxies can't make these decisions either. They don't know whether a request needs tool calling, which models to exclude for image inputs, or whether a candidate has already exceeded its cost budget or is in a cooldown period. LLMRix Model Router sits at this boundary. It's a multi-model routing and orchestration framework for Java that extracts provider differences, model selection, failover, quotas, costs, and observability out of business code. Positioned between AI applications and model services, it handles runtime decision-making and request forwarding. A chat SDK answers how to call; a Router also decides who to call, when to retry, and how to reconcile state. 1. Architecture Overview: Separating Decisions, Execution, and Infrastructure The project's architectural tradeoff is clear: routing strategies can change, but the correctness of request execution must not. To achieve this, LLMRix divides the system into layers with well-defined boundaries: - Access Layer: Provides an embedded Java API, the Orion remote client, and an OpenAI-compatible HTTP/SSE interface under Spring Boot. - Unified Contract Layer: Defines model interfaces (Chat, Embedding, Rerank, Audio, Image, Video), request/response objects, and exception types in llmrix-model-open . - Routing Core Layer: llmrix-model-router-core handles model targets, capability matching, strategy selection, execution budgets, timeouts, retries, health, and lifecycle events. - Integration Layer: llmrix-model-router-integrations provides adapters for OpenAI, DeepSeek, OpenRouter, Ollama, Redis, Bucket4j, ONNX, Shadow, evaluation, and Fugu. - State & Observability Layer: State can live in local memory/Caffeine or in Redis; events are exposed through Listeners, and Spring integration connects them to Micrometer, Observation, and Actuator. - Infrastructure Layer: TLS, WAF, external load balancing, secret management, Redis HA, and container orchestration are left to the deployment environment. Vector source: llmrix-router-architecture.svg. Original panoramic diagram: GitHub architecture SVG. The Router Core in the middle sits at the junction of decision and execution. It doesn't bind to any single vendor, nor does it rewrite business requests into provider-private objects-instead, it manages candidates through a unified ModelClient and ModelTarget . When adding a new provider, changes stay concentrated in the Provider SPI and Transport adapters, without spreading to routing strategies or business code. 1.1 Production Modules and Responsibilities | Maven Artifact | Responsibility | Typical Usage | |---|---|---| llmrix-model-open | Shared model contracts, common exceptions, auth SPI, OpenAI-compatible transport & adapters | Reuse unified types when building clients or custom integrations | llmrix-model-router-core | Router Builder, model targets, strategies, executor, state SPI, quota, health & events | Embed Router in plain Java applications | llmrix-model-router-integrations | Built-in providers, Redis, Bucket4j, ONNX, Shadow, evaluation & Fugu | Use official integrations or advanced routing capabilities | llmrix-model-router-spring-starter | Auto-configuration, YAML properties, HTTP/SSE, auth, Actuator & Micrometer | Build Spring Boot routing services | llmrix-model-orion | Framework-neutral remote Java client | Call a standalone Router from Java services | llmrix-model-orion-spring-starter | Orion auto-configuration & Micrometer adapter | Inject remote model clients in Spring Boot business services | These module boundaries let teams pull in only what they need. Remote business services can depend solely on Orion and the shared model contracts, without dragging in Redis, ONNX, or the entire Router Runtime. 1.2 Request Decision Sequence The sequence diagram below follows the source code's execution order, covering capability matching, quota control, and failover. It highlights three distinct paths: successful return, retry before the first visible output, and no-replay after streaming has begun. Vector source: llmrix-router-request-decision-sequence.svg. "Whether a model is suitable for this request" and "whether this call will succeed" are handled by two separate components: candidate snapshots and strategies handle the former, while the executor, state store, and exception classification handle the latter. 2. Feature Matrix: Model Selection and Runtime Governance | Capability Domain | Specific Features | Problems Solved | |---|---|---| | Model abstraction | Provider-neutral ModelClient , multimodal request/response, unified exceptions | Business code no longer binds to vendor SDKs | | Capability matching | operations , features , input-modalities , traits | Prevents sending tool, image, or audio requests to unsupported models | | Routing strategies | priority, round-robin, weighted random, least-busy, latency-aware, cost-aware, balanced, cache-aware | Trade off stability, cost, latency, and cache hits according to business goals | | Dynamic decisions | semantic scoring, contextual bandit, customizable RoutingStrategy | Continuously improve model selection using request semantics or feedback data | | Reliability | per-attempt timeout, total budget, retry predicates, failure thresholds, target cooldown, candidate pool continuation | Handles rate limits, timeouts, transient failures, and partial outages | | Streaming safety | first-token timeout, stream idle timeout, pre-first-chunk switching, cancellation propagation, tool request non-replay | Prevents duplicate output and replay of side-effecting tool calls | | Cost governance | input/output/cache/inference token pricing, per-request maxCostUsd , route-level RPM/TPM | Makes cost constraints part of the decision, not a post-hoc statistic | | Quota & concurrency | target-level limits, route-level limits, auth quota partitions, local Caffeine, Redis atomic leases | Controls resource usage per model, per route, and per tenant | | Multimodal | Chat, Responses core subset, Embeddings, Rerank, Audio, Image, Video | Unified handling of text, image, audio, file, and video workflows | | Evaluation & orchestration | Online Shadow, offline Evaluation, Fugu Worker/Thinker/Verifier, ONNX policies | Compare models and organize multi-round collaboration without affecting main traffic | | Observability | Router/Fugu Listener, Micrometer, Observation, Actuator, request ID | Answers "who was selected, how long it took, why it retried, and how much it cost" | | Extensibility | ModelProvider , ProviderAuthenticator , ModelPricingResolver , RouterStateStore | Integrate enterprise proxies, signed auth, internal pricing catalogs, and custom state systems | 2.1 Why Model Capabilities Are Declared in Four Categories LLMRix doesn't reduce "model capability" to a single boolean. Instead, configuration declares four separate dimensions: - operations : what the model can do, e.g.chat ,embeddings ,rerank ,video-generation ; - features : what protocol features it supports, e.g.streaming ,tools ,structured-output ,prompt-cache ; - input-modalities : what inputs it accepts, e.g.vision ,video ,audio ,file ; - traits : what task types it excels at, e.g.code ,reasoning ,long-context . The four declarations are independent. They're validated at startup and used to filter candidates per request. Compared to maintaining a single "universal model list," this approach is easier to audit and reduces capability mismatches in production. 3. From "Calling a Model" to "Calling a Route" LLMRix lets applications depend on stable route names rather than upstream model names. For example, business code simply requests general , code , or reasoning . Whether the request ultimately goes to OpenAI, DeepSeek, OpenRouter, or a local Ollama instance is decided by the router based on capability, health, latency, cost, and strategy. Model adjustments therefore live in routing configuration, not in business code. When adding providers, replacing models, or changing selection strategies, upstream applications typically don't need to be rewritten. LLMRix offers three integration modes: - As an embedded Java SDK, dropped directly into an existing application process; - As a Spring Boot Starter, auto-assembled through configuration; - As a standalone OpenAI-compatible service, providing a unified HTTP interface for other languages and existing tools. When a Java service needs to call a remote Router, it can use the project's lightweight client, Orion. Provider keys stay on the Router side only-clients only see route names and the unified protocol. 4. How a Request Is Processed LLMRix doesn't just work off a flat list of models-it processes requests through a well-bounded execution pipeline. Upon receiving a request, the Router first checks the operation type, tool calling