An Open Field Guide to Inference Engineering

An Open Field Guide to Inference Engineering

This guide is for platform engineers new to inference. If you need to understand how models and GPUs work together and how to run inference in production, this guide will help. You'll also learn about the tools to use and how to grow into an inference engineering role.

This guide builds on the work of others in the inference community. If you're interested in inference engineering, continue your learning with these:

• Viktor Farcic's LLM Inference Explained
• Philip Kiely's Inference Engineering
• The Pragmatic Engineer, What Is Inference Engineering?
• ByteByteGo's Guide to AI Inference Engineering
• Data Gravity on The Rise of Inference Engineering
• Spheron's Inference Engineering Guide
• Vishakha Sadhwani's The AI Inference Engineering Roadmap
• Upbound, Project Champagne series

1. Why now?

In an AI-forward organization, platform teams become inference teams. Coding assistants can't send source code to third parties, and frontier model bills increase month to month. Support tools also need internal data that a hosted model can't reach.

Organizations want to run these models themselves, and open models now make that practical. Hugging Face hosts more than two million open models, and the best of them compete with closed models. DeepSeek released R1 in January 2025, and it matched OpenAI's o1 on math and coding benchmarks about a month after o1 came out.

When you run your own model, you control latency, uptime, and where your data goes. At high volume, you can also cut costs by about 80%.

The work suits platform engineers. The price of a fixed level of AI capability falls roughly 10x a year, and most of that comes from how models are served, not from better GPUs.

Upbound, the company behind Crossplane and Modelplane, saw this firsthand. In its Project Champagne blog series, the team ran its own models on Modelplane and found that vLLM and scheduling worked. The time went to TLS, identity, timeouts, storage, IAM and observability.

Skills that carry over

An inference engineer picks a model, gets it onto the right GPU, keeps it serving, makes it faster and cheaper, and gets paged when it breaks. That's SRE for a new workload, and most of your skills carry over:

A hosted API is still the right call early in a product's life. Self-hosting pays off when API spend becomes a real line item, when latency or reliability needs outgrow vendors, or when data can't leave your network.

In August 2026, Chris Zeoli counted about 58 open inference roles on public careers boards, with no settled job title.

2. Inference 101

This section covers the concepts in roughly the order you'll meet them. Most are about what the engine keeps in GPU memory or what limits its speed. Viktor Farcic suggests asking when each concept starts to matter. Some matter as soon as you deploy, others only once many people use the model, and some you may never need. The table near the end of this section gives the signal for each.

What an inference engine is

An inference engine, like vLLM or SGLang, is the server that turns model weights into an API. For a platform engineer, the closest mental model is a stateful database server with a GPU attached. It loads a large dataset into memory at startup and keeps per-session state. It also schedules work internally and answers over HTTP. Ollama on a laptop is the same idea at small scale, and every hosted model API runs one behind the scenes.

Unlike a database, the engine keeps its working set in GPU memory, not RAM. GPU memory is small and expensive, so most tuning is about deciding how much of it each request gets.

The inference stack

The engine is one layer of four. At the bottom, kernels do the arithmetic. You rarely choose them directly. Above them sits the engine. Above that, a control plane deploys and manages engines, because sooner or later you run more than one. On top, a gateway routes traffic across models and replicas.

Inference stack: four layers and the clients they serve

Requests pass through the gateway to an engine, which runs kernels on the GPU. The control plane deploys and manages the gateway and engines, but no request passes through it.

Tokens

A token is the unit a language model works in, roughly a word or word fragment. Context windows, prices and speeds are all counted in tokens. When someone quotes tokens per second, they mean this unit.

Weights and the KV cache

The weights are fixed, and the KV cache grows with load. Size for both, as you would a database's footprint and working set.

Weights are the fixed footprint. Most models are released at 2 bytes per parameter (BF16 or FP16), so an 8B model needs about 16 GB, and it has to live in GPU memory, not system RAM. On a 48 GB L40S, that leaves about two-thirds of the card for everything else. That number is the same for one user or a hundred.

The KV cache is the working set. Each new token is computed against every token before it. Rather than recompute that history, the engine keeps the attention keys and values for every token in flight. Its size scales with context length multiplied by the number of concurrent conversations, much like a session store that grows with both session size and user count.

You can estimate the KV cache per token from a model's config:

KV bytes per token = 2 × layers × KV heads × head dimension × bytes per value

For Llama 3 70B at FP16 (80 layers, 8 KV heads, head dimension 128), that is about 320 KB per token. One 128K-token conversation needs about 40 GB of cache. At that size, the KV cache usually limits how many users a GPU can serve.

Watch for these sizing mistakes:

• Mixture-of-experts (MoE) models keep every weight resident. A model with 320B total parameters and 18B active still needs about 320 GB at FP8. Only compute scales with the active parameters.
• Read the checkpoint size, not the dtype table. While self-hosting its own models for Project Champagne, Upbound estimated one MoE model at about 315 GB from its dtypes (Part 2). The real checkpoint was 177.7 GB, because the experts were stored in a 4-bit format.
• Leave room for runtime overhead. CUDA graphs, activations and memory fragmentation all take space. A model that fits at startup can still run out of memory days later (Project Champagne, Part 2).

Two phases, two bottlenecks

Each request runs in two phases on the same GPU, and they are limited by different things.

Generating one token does one or two floating-point operations per byte read from memory, depending on weight precision. An H100 needs about 295 operations per byte to stay busy. That means one request uses a small fraction of the chip while the compute units wait on memory (Data Gravity).

Peak compute has grown about 3x every two years, against about 1.6x for memory bandwidth. Nearly every optimization either moves fewer bytes per token or spreads each byte across more tokens.

Because the phases have opposite bottlenecks, a technique that speeds one often does little for the other. That is why benchmarks report TTFT and tokens per second separately.

Metrics to track

Cost per million tokens converts a GPU price and a throughput into one comparable number:

Cost per million tokens = GPU cost per hour / (tokens per second × 3600) × 1,000,000

Spheron's worked example: a GPU at $2.90 an hour sustaining 1,200 tokens per second costs about $0.67 per million tokens.

Latency versus throughput

Batching more requests together raises total throughput and slows each individual user. Interactive chat and coding favor latency. Overnight batch jobs favor throughput. It's the same trade-off as tuning a database for transactions or for analytics.

Write down the workload's constraints before you tune anything.

When to adopt each concept

Each row gives the symptom that tells you it's time to act, in roughly the order teams need them.

Most teams need only the first few rows on day one. Add the rest when their signals show up.

What agents change

Coding agents and other agentic workloads stress every part of this picture. Upbound measured this in Part 5 of its Project Champagne series (introduced in Section 1), where about 10 of its engineers ran Claude Code and other coding agents against a self-hosted model. Their traffic looked like this:

• Prompts are long. The median prompt was 59K tokens and the p90 was 125K.
• Prompts are mostly repeated. About two-thirds of prompt tokens hit the prefix cache, because agents resend their transcript on every turn.
• Thinking is hidden. A reasoning model can think for more than a minute before its first visible token. Every timeout between the agent and the model has to allow for that.
• Load depends on the client. Short-prompt sessions launch subagents and can keep more requests in flight than long-prompt sessions. Plan capacity in requests in flight rather than seats.
• Sessions run for hours. Streams, tool calls and credentials all have to keep working for that long.

3. Running inference in production

In production, every choice trades latency, throughput, cost and quality against each other, usually with too few GPUs. Where a practice comes from Project Champagne, Section 5 has the details.

3.1 Start from use cases, not hardware

Each use case sets different requirements. In Project Champagne:

Coding agents needed long context without quality loss, reliable