You Don't Need to Choose Between a Gateway and an Agent Framework

When I first published Swarm on GitHub, most questions weren't about Rust or MCP. They were about timing and categorization: "We just need a lightweight gateway for multi-provider routing; agents feel like overkill." "We already run an orchestration framework; why would we replace our proxy?" This reaction highlights a false dichotomy currently plaguing the AI infrastructure ecosystem: the assumption that a gateway and an agent orchestrator must be two completely different products. In practice, teams rarely wake up needing full-blown multi-agent autonomous swarms on Day 1. But when they start with a standalone proxy, they inevitably hit a wall - patching together Python microservices, external vector state stores, MCP bridges, and ad-hoc eval scripts. Every evolution requires a rewrite. The core premise of Swarm is different: a single, pure-Rust runtime where you don't choose between a gateway and an orchestrator - you simply choose which capabilities to turn on. The AI Adoption Ladder Most engineering teams evolve their LLM stack along a predictable trajectory: Rung 1: OpenAI-Compatible Gateway (Drop-in replacement for hardcoded SDKs) └── Rung 2: Multi-Provider Fallbacks (Groq, Gemini, Ollama, vLLM via TOML) └── Rung 3: Stateful Sessions (Previous response chaining & context) └── Rung 4: Native MCP Tools (SSE + Streamable HTTP tool execution) └── Rung 5: Multi-Agent DAGs (Planner + Executor + Specialists) └── Rung 6: Built-in Evals (LLM-as-a-Judge & policy gates) You can stop at any rung and have a lean, production-grade binary. When you're ready for the next level, you change a configuration flag - not your architectural foundation. Rung 1 - Just a Low-Latency Gateway If your immediate goal is simply eliminating hardcoded API keys and single-vendor SDK locks, Swarm acts as an OpenAI-compatible drop-in front door with sub-millisecond native routing overhead. # Spin up the gateway in seconds ./kickstart/gateway_kickstart/01_launch_gateway.sh curl -X POST http://localhost:8080/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "openai/gpt-4o-mini", "messages": [{"role": "user", "content": "Explain progressive disclosure in software."}] }' You get instant OpenAI compatibility. No agent overhead, no background worker queues, no forced abstractions. Rung 2 - Multi-Provider & Local Model Routing When rate limits hit or you need cost-effective fallbacks across cloud and local runtimes (Groq, Anthropic, Gemini, Ollama, vLLM, llama.cpp), routing is declared cleanly in config.toml : [providers.groq] api_url = "https://api.groq.com/openai/v1/chat/completions" weight = 80 [providers.local_vllm] api_url = "http://localhost:8000/v1/chat/completions" recommended_models = ["meta-llama/Llama-3.3-70B-Instruct"] Your applications continue calling the same /v1/chat/completions endpoint. Failover, load distribution, and local-inference routing happen invisibly inside the runtime. Rung 3 - Stateful Conversations via /v1/responses Multi-turn chat state is where teams often bolt on an external Redis or PostgreSQL session manager. Swarm provides explicit turn-by-turn state management natively through /v1/responses using previous_response_id chaining: curl -X POST http://localhost:8080/v1/responses \ -H "Content-Type: application/json" \ -d '{ "model": "groq/llama-3.3-70b-versatile", "input": "Calculate the Q3 cloud infrastructure spend.", "previous_response_id": "resp_01JMW892KPA7XYZ" }' State is managed by the runtime, eliminating client-side conversation bloat while keeping state inspection simple and deterministic. Rung 4 - Native Model Context Protocol (MCP) When your model needs real-world context - database schemas, filesystem access, API calls - you shouldn't have to migrate to a heavy agent framework just to call tools. Swarm natively supports MCP (over both SSE and streamable HTTP) directly inside the gateway layer: [mcp_servers.postgres_db] transport = "sse" url = "http://localhost:3001/sse" [mcp_servers.git_tools] transport = "http" url = "http://localhost:3002/mcp" Tool discovery, argument validation, and streaming tool execution run within the same engine that routes your completions. Rung 5 - Coordinated Multi-Agent Workflows When single-prompt loops cannot solve compound tasks, Swarm activates its autonomous orchestration engine: User Intent │ ▼ [ Planner ] ──► Builds Execution DAG (Dependencies & Concurrency) │ ▼ [ Executor ] ──► Dispatches tasks across Domain Specialists │ ├── Specialist A (Data Analyst + Postgres MCP) └── Specialist B (Report Writer + File MCP) │ ▼ Unified Response Why this matters: Rung 5 reuses the identical provider configurations, fallback pools, state engine, and MCP tool connectors established in Rungs 1-4. There is no secondary agent daemon or translation bridge. Rung 6 - Built-in LLM-as-a-Judge Evaluation The final rung is the one most gateways and agent frameworks omit entirely: closing the loop on quality. Instead of exporting logs to an external SaaS pipeline, Swarm embeds an LLM-as-a-Judge loop. It scores intermediate DAG outputs, validates MCP tool results against deterministic schemas, and flags hallucinated responses before they reach client applications: curl -X POST http://localhost:8080/v1/eval/judge \ -H "Content-Type: application/json" \ -d '{ "response_id": "resp_01JMW892KPA7XYZ", "criteria": ["correctness", "grounding", "conciseness"], "judge_model": "openai/gpt-4o" }' This foundational layer enables our upcoming roadmap items: policy-based dynamic routing, durable state checkpoints, and human-in-the-loop validation gates. Architectural Coherence Beats Glue Code The individual capabilities of Swarm - gateway proxying, MCP tool invocation, DAG planning, automated evaluation - exist across different open-source projects. What is rare is finding them integrated into a single, zero-dependency, memory-safe binary where adopting multi-agent orchestration doesn't invalidate the proxy architecture you set up on Day 1. The architectural bet of Swarm is simple: the tools you choose when you only need a gateway should never become technical debt the day you need agents. Discussion If you're currently scaling your LLM infrastructure: - At which rung on this ladder has your team spent the most engineering time? - Have you had to replace a gateway when moving to agents (or vice-versa)? Check out the project and try the kickstart scripts on GitHub: github.com/fcn06/swarm Top comments (0)