Designing an API Gateway: Routing, Auth, and the Filter Chain

Once you split a monolith into services, a new problem appears: the client now needs to know about twenty endpoints, each with its own auth, its own retry logic, and its own rate limits. An API gateway is the single front door that hides all of that.

The problem it solves

Every service needs the same cross-cutting concerns. Authentication, TLS termination, rate limiting, request logging, and CORS all have to happen somewhere. If each service implements them, you get twenty slightly different, slightly buggy copies. The gateway pulls these concerns out of the services and does them once, in one place, before the request is routed anywhere.

It also gives clients a stable contract. The mobile app talks to api.example.com and does not care that the orders service moved, split in two, or got rewritten in a different language.

Routing

The core job is mapping an incoming request to a backend. A request to /orders/123 needs to reach the orders service; /users/me reaches the user service. Routes are usually matched by path prefix, sometimes by host header or HTTP method. The gateway holds a route table, and modern gateways refresh this table dynamically from service discovery so you are not redeploying the gateway every time a backend changes address.

A useful pattern here is the backend for frontend. A web client and a mobile client often want differently shaped responses. Instead of one bloated API, you run a thin gateway per client type, each aggregating the same services but tailoring the payload.

The filter chain

The cleanest way to structure a gateway is as an ordered chain of filters that every request passes through. Each filter does one thing and either passes the request along or rejects it.

A typical inbound chain looks like this:

• TLS termination and request parsing
• Authentication: validate the token, reject if invalid
• Authorization: does this identity have access to this route
• Rate limiting: has this client exceeded its quota
• Request transformation: add headers, strip internal fields
• Routing: pick the backend and forward

On the way back, a smaller outbound chain runs: response transformation, adding CORS headers, and logging the status and latency.

The value of this design is that each concern is isolated and ordered. Auth must run before rate limiting so you are counting against the right identity, and both must run before you spend resources forwarding to a backend.

Auth: where to draw the line

The common approach is token-based. The client sends a JWT or an opaque token, and the gateway validates it. A JWT can be verified locally using a public key, which is fast because there is no network call, but revoking one before it expires is hard. An opaque token requires a lookup against an auth service or cache on every request, which is slower but gives you instant revocation. Many teams use short-lived JWTs to get the speed while capping the blast radius of a leaked token.

Crucially, the gateway authenticates but usually does not do fine-grained authorization. It confirms who you are and passes a trusted identity header to the backend, which decides whether you can touch this specific resource. Coarse checks at the edge, fine checks at the service.

The trade-off you cannot ignore

A gateway is a single point of failure and a potential bottleneck. Every request in your system flows through it, so it must be horizontally scaled and stateless, with any shared state (like rate limit counters) pushed to Redis. Keep the filters cheap. Heavy work such as response aggregation across many services adds latency to every call, so measure it. The gateway should add single-digit milliseconds, not tens.

How the real systems do it

Netflix built Zuul around exactly this filter-chain model, with pre, route, and post filters. Kong runs as a set of Lua plugins on top of Nginx, where each plugin is a filter you enable per route. Envoy, the data plane behind many service meshes, expresses the same idea as a chain of HTTP filters configured dynamically.

The shape is always the same: an ordered pipeline, cross-cutting concerns pulled forward, and a stable front door for clients.

I wrote the full breakdown, with diagrams and the data model, here: https://www.systemdesign.academy/interview/design-api-gateway