DEV Community

Per-user AI token quotas: your backend decides who is counted, and a 429 is not a diagnosis

A per-user token quota rests on three decisions: who the user is, what you count, and who answers when the limit is reached. The first one decides whether the other two mean anything.

If the browser or the mobile app can set the user id or the plan name, a caller can change whose quota is charged, and the quota holds only for users who don't try.

Two claims run through this article. First, the gateway counts, but your backend decides who is counted. Authentication and the entitlement lookup belong on your side of the trust boundary, and whatever the gateway receives is an assertion from whoever holds the API key.

Second, a 429 tells you that someone refused, not who refused or why. A provider rate limit, a per-user quota and your gateway vendor's own plan limit can all arrive as 429, and each needs a different response from your client.

Everything below that describes expected behavior is a design criterion, not a result from tests we ran. The user ids, plans and limits are fictional.

The gateway counts. Your backend decides who is counted.

The user identity at the gateway is only as trustworthy as the component that set it. There are two sound patterns.

Your backend sets it. The user authenticates to your app. Your backend reads the user id from the verified session, looks up their plan in your billing system, and attaches both to the model call. Any identity or plan header that arrives from the client is dropped, never forwarded.

An identity-aware proxy sets it. Cloudflare's AI Gateway behind Cloudflare Access is one documented example: when a request carries a valid Access JWT, the gateway records the JWT sub claim as cf.user_id. The same page lists the limit of that model: service-token requests carry no cf.user_id, because a service token is not a person.

Batch jobs and agents acting for a user have the same gap in every design. Something has to attribute them explicitly.

The pattern to avoid is a client-set header passed through untouched. A gateway can only act on the identity it is handed. If the client controls that header, the gateway enforces the quota faithfully against whatever name the caller chose.

Send a pseudonym, not an email address. Derive it with a keyed hash so nobody can recompute it from a user id alone:

import hashlib
import hmac
import os
from openai import OpenAI

client = OpenAI(
    base_url="https://gateway.worksi.ai/v1",
    api_key=os.environ["AIW_API_KEY"],  # server-side only
)

PSEUDONYM_KEY = os.environ["END_USER_PSEUDONYM_KEY"].encode()

def end_user_id(user_id: str) -> str:
    return hmac.new(PSEUDONYM_KEY, user_id.encode(), hashlib.sha256).hexdigest()[:32]

def call_model(session, entitlements, messages):
    # session.user_id comes from your verified session,
    # never from the request body or a client header.
    plan = entitlements.plan_for(session.user_id)  # your billing system
    return client.chat.completions.create(
        model="gpt-5-mini",
        messages=messages,
        extra_headers={
            "X-AIW-End-User": end_user_id(session.user_id),
            "X-AIW-End-User-Plan": plan,
        },
    )

Two consequences follow. A pseudonym is an identifier, not proof of identity: anyone holding the gateway API key can send any value, so the key never ships in a browser bundle or a mobile binary.

And rotating the hash key changes every user's identifier, so usage before and after the rotation lands on two different ids. Treat the key as long-lived, and before any rotation find out how your implementation handles a changed identifier for each quota dimension.

Decision table: identity and quota dimensions

  • Identity source
    • Decision Options: Verified session in your backend; identity-aware proxy subject; client-set header
    • Choose by: Only the first two are trust boundaries
    • What breaks if you choose wrong: Anyone can spend anyone's quota, or claim a higher plan
  • Who is counted
    • Decision Options: Individual user; account or tenant; both
    • Choose by: Per-seat pricing points to the user, pooled plans to the account, both when one member must not drain a shared pool
    • What breaks if you choose wrong: Per-account only: one heavy member exhausts everyone. Per-user only: a tenant multiplies its allowance by adding seats
  • Requests per window
    • Decision Options: Per minute, per day
    • Choose by: Stops loops and high-frequency abuse
    • What breaks if you choose wrong: One 100,000-token call counts the same as a 50-token call
  • Tokens
    • Decision Options: Input, output, total; cached input counted or not
    • Choose by: Bounds model work per user
    • What breaks if you choose wrong: Input and output can be priced differently, and providers report cached input separately
  • Model eligibility
    • Decision Options: Allowlist per plan
    • Choose by: Product tiers
    • What breaks if you choose wrong: The refusal may use a different code from the quota, so the client mishandles it
  • Spend
    • Decision Options: Currency, window, source of prices
    • Choose by: What finance and your margin care about
    • What breaks if you choose wrong: The figure is an estimate from a price list, not the invoice

Combining scopes is a documented pattern elsewhere. Octelium's AI gateway enforces every matching token limit, which lets one service carry both a per-user and a per-tenant quota, and it recommends a separate request rate limit to catch high-frequency loops.

Tokens, requests and spend measure different things

A token limit is not a spend limit. Anthropic's pricing page, read on 5 November 2026, lists Claude Haiku 4.5 at $1 per million input tokens and $5 per million output tokens, and Claude Sonnet 4.6 at $3 and $15. The same 20,000 tokens cost different amounts depending on the model and on the input and output split. A per-user token cap on a plan that allows several models bounds work, not money.

Cached input makes the gap wider. Anthropic's rate limit documentation explains that input_tokens covers only the tokens after the last cache breakpoint. Its example: a 200k-token cached document and a 50-token question report input_tokens: 50, with the cached part in cache_read_input_tokens. A quota that sums input_tokens lets that user through on almost nothing. A quota that also counts cache_read_input_tokens charges them in full for tokens the same pricing page, read on 5 November 2026, lists at 0.1x the base input rate. Decide which you mean, and find out which fields your gateway reads.

Spend limits have their own catch. Traefik's token quota middleware offers separate input, output and total token limits plus a cost limit in US dollars. Its docs label that cost as an estimate from provider-reported tokens and the middleware's own price list, and send you to the provider's billing dashboard for the real amount. Any per-user spend figure works the same way: the price list is part of the quota, and a stale price is a wrong limit.

In-flight requests: reserve first or settle after

You cannot know a request's token count until the response finishes, so there are two designs, and each leaks differently.

Reserve, then reconcile. Before calling the provider, the gateway holds an estimate: input tokens plus the maximum output the request allows. If the reservation does not fit, the request is refused. After the response, the hold is replaced by reported usage. Octelium documents this design: its token limit "reserves estimated input and maximum output capacity before calling the provider, then reconciles it with provider-reported usage." The cost is false refusals. A user with 3,000 tokens left who sends max_tokens: 4096 is refused even if the answer would have been 200 tokens.

Count after the response. The request goes through if the user is under the limit now, and usage is added when it completes. App AI Gateway documents the consequence for its budgets: cost is known only when the response completes, the counter reads the month's spend at most every ten seconds, and "a single very expensive call, or a burst inside those seconds, can overshoot."

Neither design is wrong. Reservation suits hard caps on expensive plans. Post-response accounting suits generous plans where a small overshoot is cheaper than a false refusal. Testing tells you which one you run.

Then decide what happens when usage never arrives. A client cancels a stream halfway. An upstream times out. A response format carries no usage at all: Traefik's docs say MCP responses carry no token usage data, and the middleware rejects token rate limiting for them rather than guessing. If an implementation records missing usage as zero, the quota undercounts. Test which of four things your implementation does when usage is absent: reconcile later, charge an estimate, keep the reservation, or record zero. Our recommendation under reservation is to keep the hold until real usage replaces it.

Four kinds of 429, and two refusals that are not 429

RFC 6585 defines 429 as "too many requests in a given amount of time", says the response MAY include Retry-After, and leaves open how the server identifies the user and counts requests. So the status code carries almost no diagnosis. These are the causes to separate, using Worksi's codes as the gateway example:

  • Provider 429, rate limit
    • Who refused: Your provider, for your whole organization or project
    • What separates it: Provider error body and request id. OpenAI's docs say its limits are set at organization and project level, not user level, so it says nothing about which of your users caused it
    • What the client should do: Back off. Never show this user an upgrade prompt
  • Provider 429, spend cap
    • Who refused: Your provider
    • What separates it: Anthropic returns rate_limit_error, the same type as a rate limit, with error_code enforced_spend_limit_reached and no retry-after header. Retrying fails until access resumes
    • What the client should do: Stop retrying and alert your own team
  • 429 end_user_quota_exceeded
    • Who refused: Worksi, for this end user
    • What separates it: The call never reached the provider. The end-user plans docs say Retry-After gives the window reset and the error details name the plan, the limit and the window
    • What the client should do: Show the user their limit and an upgrade or try-later message
  • 429 plan_quota_exceeded
    • Who refused: Worksi, for your organization's subscription
    • What separates it: Read details.limit. When the monthly AI-traces quota is used up, new traces are refused while calls in traces already admitted continue, until the next UTC month or a larger plan. A fair-use backstop on gateway requests can return the same code
    • What the client should do: Alert your team. It is not the user's fault
  • 403 model_not_allowed
    • Who refused: Worksi model allowlist, including a plan's models list
    • What separates it: Not a 429 at all
    • What the client should do: Fall back to a model the plan allows
  • 402 budget_exceeded
    • Who refused: Worksi spend policy, organization or narrower scope
    • What separates it: Not a 429 at all
    • What the client should do: Alert your team

Do not build one retry schedule for all of them. A client that retries every 429 with backoff hammers a spend cap that will not lift for weeks, and tells a quota-limited user to wait when they need an upgrade.

The test matrix

Run these in a staging environment with fictional users: u_free_a on plan free with a fictional 20,000 tokens a day, and u_pro_b on plan pro. Expected results follow this article's design, which is a recommendation, not a standard. Where your implementation differs, record what it does.

  1. Two users, two plans
    • What is checked: Limits are per user and per plan
    • Expected under this design: After u_free_a hits its limit, u_pro_b continues unaffected
    • Evidence that separates causes: Per-user usage shows two ids. The refusal names plan free
  2. Spoofed identity
    • What is checked: The backend strips client headers
    • Expected under this design: A browser request carrying X-AIW-End-User for u_pro_b and plan pro is charged to u_free_a on free
    • Evidence that separates causes: The end-user id recorded at the gateway equals the HMAC of u_free_a
  3. Missing identity
    • What is checked: Unattributed calls are refused or attributed on purpose
    • Expected under this design: Unauthenticated requests get your own 401 before any model call. Background jobs carry an explicit system id with its own plan
    • Evidence that separates causes: No gateway record without an end-user id
  4. Direct provider bypass
    • What is checked: No path to the provider skips the quota
    • Expected under this design: No service outside the gateway holds a provider key
    • Evidence that separates causes: An unmatched provider request or credential is a lead. A bypass is shown only by an application or egress log of a direct provider call, matched to the provider request id
  5. Per-user vs organization-wide
    • What is checked: Shared limits and user limits are distinct
    • Expected under this design: Many users each under quota can still trip a provider limit (429) or an organization spend policy (402), and get no upgrade prompt
    • Evidence that separates causes: Error code and origin differ from the exhausted-quota case
  6. Exhausted user quota
    • What is checked: The refusal happens before the provider
    • Expected under this design: 429 end_user_quota_exceeded with Retry-After
    • Evidence that separates causes: Provider usage does not move. The gateway records a refusal
  7. Upstream rate limit
    • What is checked: Provider 429 is not mistaken for a quota
    • Expected under this design: The user's quota is unchanged and the client backs off
    • Evidence that separates causes: Provider error body and request id. To force it, point a test route at a stub upstream you control that returns 429
  8. Concurrency near the limit
    • What is checked: Reservation or overshoot
    • Expected under this design: u_free_a has 1,000 tokens left and sends ten parallel calls with max_tokens: 500. Reservation permits at most two outstanding holds at once, fewer if the prompts' input estimates count against the remaining 1,000. Post-response accounting may admit all ten
    • Evidence that separates causes: Final counter vs the limit. To test total admissions, use a stub upstream that holds every response until all ten admission checks complete, so no early reconciliation frees capacity
  9. Missing usage
    • What is checked: Cancelled or usage-less calls are still charged
    • Expected under this design: Cancel a stream after the first chunk. Under a reservation design that keeps the hold, the counter moves by at least the reservation
    • Evidence that separates causes: Counter delta for the call, and which of reconcile, estimate, hold or zero happened

The bypass row is the easiest to get wrong in both directions. Any service with a raw provider key, a second gateway route without the quota, or a forgotten staging key can call the model without being counted. But a mismatch between the provider's usage and the gateway's totals has innocent causes too. Before you treat it as a bypass, align the comparison:

  • Metric and fields. Compare the same token fields, including cached input, or compare request counts, not tokens on one side and requests on the other.
  • Window. Use the same time zone and boundaries, and wait until both the provider's usage reporting and the gateway's telemetry have caught up.
  • Telemetry completeness. Confirm the gateway's records for the wind
Read on DEV Community ↗ ← Back to News

Comments

No comments yet. Start the discussion.