6 AI Gateways Compared for 2026: Routing, Governance, Caching, and Observability

TL;DR An AI gateway is one control plane for every LLM request your application makes, regardless of which provider serves it. Without one, the pieces scatter: provider keys in one place, rate limits in another, caching somewhere else, and separate homes for permissions, spend reporting, and logs. That sprawl gets harder to govern with every new service that starts calling a model. The things worth evaluating are fairly concrete - does it consolidate provider access, cap spend, deduplicate calls with caching, keep durable audit records, and let you see what production is actually doing when latency, cost, or output quality moves. This post compares six gateways across provider coverage, governance, caching, cost visibility, audit logging, and observability. They break down into two rough camps, and the right pick depends mostly on which constraint is actually binding for you.

What an AI gateway does

An AI gateway sits between your application code and your model providers. Every LLM request passes through it on the way out. Without that layer, each service handles its own provider keys, retries, rate limits, caching, and logging. With it, services hit a single endpoint that applies one consistent set of access rules, quotas, cost accounting, and audit records across every provider behind it.

"AI gateway" and "LLM gateway" generally describe the same thing. The first term shows up more when the conversation is about governing production; the second when it's about developers routing between models. Model routing and a unified API are features of a gateway, not separate product categories - worth knowing when you're reading vendor pages.

Most options fall into one of two camps:

• Infrastructure-first gateways extend an existing API management, Kubernetes, or edge platform to cover AI traffic. You inherit the deployment model, the policy engine, and the operational overhead you already have.
• LLM-native gateways start from model access, prompt traffic, observability, and provider abstraction. You get AI-specific features first and general-purpose traffic management second.

Neither is better in the abstract. Pick based on whether your real constraint is infrastructure control, provider routing, or tying production traffic to some measure of quality.

What to look for

• Provider and model breadth. Broad coverage means one endpoint reaches every model you care about, so adopting a new one doesn't mean writing another provider-specific integration and shipping it to every service.
• Rate limiting and quotas. Limits scoped per key, per team, and per model are what stop a runaway agent loop, a staging environment, or one hungry feature from quietly eating the month's budget.
• Access control and governance. Role-based permissions, virtual keys, and policy enforcement in one place give engineering, security, and finance a shared handle on model usage instead of three different partial views.
• Caching. Caching responses cuts both cost and latency on repeat calls. The details matter more than the checkbox: can you control expiration, scope, and per-request behavior, and can you tell whether a given response was served from cache?
• Cost tracking and attribution. Spend should slice by team, feature, model, user, or environment. A single org-wide number tells you that you have a problem but not where it is.
• Audit logging. Durable records make it possible to reconstruct which requests ran, who made them, what the payloads were, and how they resolved - during an incident, or during a compliance review six months later.
• Actionable observability. Production logs get considerably more useful when a trace can become an evaluation case, a scorer, or a release check, rather than just a row in a dashboard.

That last criterion is the one most comparisons skip, and it's where these tools differ most. A gateway already holds the inputs, outputs, latency, token counts, and cost for every call. Whether that data can feed an evaluation workflow - or just a metrics view - determines whether production traffic helps you catch regressions before the next prompt or model change ships, or only helps you explain them afterward.

1. Braintrust Gateway

Best for: teams shipping production AI applications who want model routing joined up with tracing and evaluation in one system.

Braintrust Gateway gives you a single API for routing across providers: OpenAI, Anthropic, Google, AWS, Azure, Mistral, and others on its supported list. Setup is about as light as it gets - point an existing SDK at https://gateway.braintrust.dev, authenticate with a Braintrust API key, and call supported models without carrying separate provider integrations through every service. Provider keys are configured at the organization or project level, and custom providers let you route to self-hosted models, fine-tuned models, and proprietary services through the same OpenAI-compatible client.

The distinguishing property is that gateway traffic lands in the observability and evaluation layer by default. Calls through the gateway are traced without extra SDK instrumentation, so inputs, outputs, latency, token usage, cost, and caching behavior are all inspectable. Those spans can then be scored, annotated with metadata or feedback, added to datasets, compared across experiments, and used in CI checks - which is a meaningfully different workflow from exporting logs and rebuilding that pipeline yourself.

The operational controls are present too. Response caching cuts repeat calls, with request headers, TTL settings, and cache-status visibility governing when a cached response gets served. Cached data is encrypted with AES-GCM under a key derived from the caller's API key, and cached results are scoped to that user by default. Response headers expose cache status, provider endpoint, error origin, request ID, and logged span ID - enough to correlate routing behavior with debugging and cost attribution.

Pros

• One endpoint across major providers, including OpenAI, Anthropic, Google, AWS, and Mistral
• Drop-in with existing SDKs, so trying a new model doesn't touch your client layer
• Provider configuration centralized at org or project level
• Custom providers for private endpoints, fine-tuned models, and self-hosted deployments
• Gateway logs connect natively to tracing, scoring, datasets, experiments, and CI/CD checks
• Per-request caching controls with TTL and visible cache status
• Cached responses encrypted and user-scoped by default
• Dashboards for usage, cost, latency, errors, and quality signals

Cons

• Hosted Gateway is still in beta
• Self-hosting requires the Enterprise plan
• Gateway-native quota controls are lighter than what infrastructure gateways offer

Pricing: Free Starter plan (1 GB processed data, 10K scores, unlimited users). Pro at $249/month. Custom enterprise pricing. The gateway itself is free during beta.

2. Portkey

Best for: enterprise teams that want a managed gateway bundled with governance, guardrails, prompt management, and compliance controls.

Portkey packages an AI Gateway alongside Observability, Guardrails, Governance, and Prompt Management. Its Universal API covers routing, fallbacks, load balancing, retries, virtual keys, logs, traces, feedback, metadata, filters, and alerts. It fits when you want gateway controls and guardrails from one managed vendor, especially where the requirements list includes RBAC, SSO, granular budget and rate limits, private cloud deployment, VPC hosting, data export, or compliance controls. Both deterministic and LLM-based guardrails are supported, which is one of the stronger parts of the product. If you gate releases on evaluation results, you'll likely still run that workflow elsewhere.

Pros

• Universal API with routing, fallbacks, load balancing, and retries
• Virtual keys, logs, traces, metadata, filters, and alerts
• Deterministic and LLM-based guardrails
• Production plan includes RBAC and service account API keys
• Enterprise covers SSO, granular budget and rate limits, private cloud, VPC hosting, and compliance options

Cons

• Free plan is scoped to prototyping, testing, and enterprise POCs
• Cost scales with recorded logs and enterprise requirements
• Evaluation is less structured than in tools built around datasets, scorers, experiments, and CI checks

Pricing: Free tier with 10K logged requests. Paid plans from $49/month. Custom enterprise pricing.

3. LiteLLM

Best for: teams with DevOps bandwidth who want an open-source gateway they run themselves.

LiteLLM ships a Python SDK and a Proxy Server that route requests to 100+ providers behind an OpenAI-compatible interface. The open-source plan includes provider integrations, virtual keys, budgets, teams, load balancing, RPM and TPM limits, guardrails, and logging integrations - a genuinely broad feature set for something you can pip install. It's the right pick if you want infrastructure control and accept the operational cost of running the gateway.

Logging integrations cover Braintrust, Langfuse, Arize Phoenix, LangSmith, and OpenTelemetry, so it composes reasonably well with whatever observability stack you already have. Enterprise adds cloud or self-hosted deployment, support, custom SLAs, JWT auth, SSO, and audit logs.

Pros

• Open source, 100+ provider integrations
• OpenAI-compatible Proxy Server plus a Python SDK
• Virtual keys, budgets, teams, load balancing, RPM/TPM limits
• Logging integrations for Braintrust, Langfuse, Arize Phoenix, LangSmith, and OpenTelemetry
• Enterprise supports both cloud and self-hosted deployment

Cons

• Running it in production means owning it in engineering
• SSO, audit logs, custom SLAs, and support sit behind Enterprise
• Evaluation and release checks live in other tools

Pricing: Free and open source for self-hosted use. Custom enterprise pricing for hosted management and enterprise features.

4. Kong AI Gateway

Best for: enterprises already running Kong who want AI controls inside the API management stack they operate today.

Kong AI Gateway layers AI-specific capabilities onto Kong Gateway and Kong Konnect via plugins: universal LLM API routing, rate limiting, semantic caching, semantic routing, MCP and A2A traffic gateway support, RAG injection, data governance, guardrails, prompt engineering, load balancing, audit logs, LLM metrics, metering, cost controls, and secrets management.

The appeal is consolidation. If your platform team already governs APIs through Kong, AI traffic can be governed by the same infrastructure, the same policies, and the same on-call rotation. The cost is that you're adopting AI features inside a full API management platform, with the setup and pricing that implies. Structured evaluation, scorer development, and CI release gates will need a separate system.

Pros

• AI controls inside Kong's existing deployment and policy model
• Universal LLM API, token-based rate limiting, semantic caching, guardrails
• MCP and A2A traffic gateway support
• LLM metrics, audit logs, metering, cost controls
• Fully self-hosted gateways available on Enterprise

Cons

• Setup and pricing carry the weight of a full API management platform
• Some AI Gateway plugins and AI Gateway Manager are paid add-ons outside the trial
• No native evaluation workflow

Pricing: 30-day free trial. Plus plan billed per gateway. Custom enterprise pricing.

5. SUSE AI Universal Proxy

Best for: Kubernetes-native enterprises that need on-prem control over MCP servers, AI services, and private AI infrastructure.

SUSE AI Universal Proxy is an open-source (Apache 2.0) project for managing and proxying MCP servers. It sits inside SUSE's broader cloud-native AI direction and focuses on discovery, registration, authentication, governance, cost tracking, logging, and MCP service control across enterprise environments.

Note the scope carefully: this is more an MCP governance layer than an LLM router. It's a strong fit for teams standardizing AI services on Kubernetes where shadow AI discovery, private deployment, and centralized MCP governance are the priorities. For plain routing across hosted model APIs, the LLM-native gateways are a more direct path.

Pros

• Kubernetes-native, aligned with SUSE Rancher and Helm-based deployment
• MCP proxy service, server registry, discovery, and authentication
• Shadow AI discovery and centralized MCP governance
• Open source under Apache 2.0
• Suits private and sovereign AI infrastructure requirements

Cons

• Oriented toward MCP service governance more than general LLM API routing
• Younger than established API gateways and LLM-native proxies
• Requires Kubernetes and platform operations capacity

Pricing: Open source, deployed on your own Kubernetes infrastructure.

6. Cloudflare AI Gateway

Best for: teams that want caching, rate limiting, and usage visibility running at the edge with nothing to operate.

Cloudflare AI Gateway routes AI requests through Cloudflare for providers including OpenAI, Anthropic, Google, Replicate, and Workers AI. The feature set covers analytics, logging, caching, rate limiting, request retry, model fallback, persistent logs with plan-specific storage limits, DLP scanning, guardrails, and unified billing.

The pitch is low friction: core features are free on all plans, and there's no proxy for you to run. Observability centers on request metrics, logs, usage analytics, and cost visibility, so structured evaluation and CI gating need a separate workflow. Some capabilities are in beta or depend on other Cloudflare subscriptions, which is worth checking against your plan before committing.

Pros

• Core gateway features on every Cloudflare plan
• Caching, rate limiting, request retry, model fallback
• Analytics for requests, tokens, and cost
• Persistent logs with plan-specific storage limits
• DLP scanning and guardrails available

Cons

• Observability leans usage-and-logs rather than evaluation
• Some capabilities are beta or tied to other Cloudflare subscriptions
• Assumes you're comfortable routing AI traffic through Cloudflare

Pricing: Core features are free on all plans.

How to pick

Infrastructure governance is the constraint? (The article ends with "Infrastructure governance is the constraint" - that appears to be a cut-off sentence. The original text ends with "Infrastructure governance is the constraint". We must preserve exactly. So we include that as a final paragraph, but note it's likely incomplete. We'll keep it as is.)

Infrastructure governance is the constraint