AI & LLM Terminology Glossary: From Tokens to Orchestration

Core model concepts

Token - The basic unit a language model reads and generates. Not quite a word, not quite a character - usually a sub-word chunk (e.g. "tokenization" might split into "token" + "ization"). Every cost, context limit, and latency figure you'll deal with is measured in tokens, not words or characters.

Context window - The maximum number of tokens (input + output combined) a model can attend to in a single request. Everything the model "knows" about the current conversation has to fit inside this window - anything older gets truncated or has to be summarized.

Inference - The act of running a trained model to produce an output, as opposed to training it. When people talk about "inference cost" or "inference latency," they mean the cost/speed of using an already-trained model, not building one.

Parameters (weights) - The learned numerical values inside a model that encode what it "knows." Model size is usually described by parameter count (e.g. "a 7B model" = 7 billion parameters). More parameters generally means more capability, but also more memory and compute to run.

Fine-tuning - Continuing to train an already-trained (pretrained) model on a narrower, task-specific dataset, so it specializes without starting from scratch.

LoRA (Low-Rank Adaptation) - A fine-tuning technique that freezes the original model weights and trains a small set of additional low-rank matrices instead. It gets most of the benefit of full fine-tuning at a fraction of the compute and memory cost, and it's why you'll see so many "fine-tuned" open-weight variants floating around - full fine-tuning is expensive, LoRA isn't.

Quantization - Reducing the numerical precision used to store model weights (or the KV cache - see below) - for example fp16 down to int8 or int4. This shrinks memory footprint and often improves throughput, at some cost to output quality depending on how aggressive the quantization is.

Attention mechanism terms

Attention - The mechanism that lets a model weigh how relevant each previous token is when producing the next one. It's the core operation that makes transformers work, and it's also the most expensive part computationally.

KV cache (Key/Value cache) - Storage of the key and value tensors computed for previous tokens, so the model doesn't have to recompute them for every new token generated. It's usually the actual memory bottleneck in production serving - not the model weights - because it grows with both sequence length and number of concurrent requests.

MHA (Multi-Head Attention) - The original attention design, where every query head has its own dedicated set of key/value heads. Best quality, but the most expensive KV cache.

GQA (Grouped-Query Attention) - Multiple query heads share a smaller set of key/value heads, shrinking the KV cache significantly (often 4-8x) with minimal quality loss. Most modern open-weight models use this by default.

MQA (Multi-Query Attention) - The extreme version of GQA: all query heads share a single key/value head. Smallest possible cache, but a larger quality tradeoff than GQA.

Prefix caching - Reusing the KV cache for a shared prompt prefix (like a system prompt or few-shot examples) across multiple requests, instead of recomputing and re-storing it every time. Valuable for agent and chat workloads where the same instructions repeat across many calls.

PagedAttention - The memory management technique (popularized by vLLM) that treats the KV cache like OS-style virtual memory pages instead of requiring one contiguous block per request. This is what makes prefix caching and high-concurrency serving practical without the cache fragmenting into unusable gaps.

Retrieval & knowledge terms

Embedding - A numerical vector representation of text (or images, audio, etc.) that captures semantic meaning, such that similar content ends up with similar vectors. This is the foundation of search and retrieval in AI systems.

Vector database - A database optimized for storing embeddings and doing fast similarity search over them (e.g. "find the 5 most similar chunks to this query"). Examples include Pinecone, Weaviate, Qdrant, and pgvector as a Postgres extension.

RAG (Retrieval-Augmented Generation) - An architecture where the model's response is grounded by first retrieving relevant documents/chunks (usually via a vector database) and injecting them into the prompt, rather than relying purely on what the model memorized during training. This is how you get a model to answer questions about your own private documents.

Chunking - Splitting source documents into smaller pieces before embedding them, since embedding an entire document as one vector loses too much granularity for precise retrieval. Chunk size and overlap strategy directly affect RAG retrieval quality.

Reranking - A second-pass step in RAG pipelines where an initial broad set of retrieved candidates gets re-scored by a more precise (often more expensive) model before the final few are used, improving precision over similarity search alone.

Agentic system terms

Agent - A system where a model doesn't just respond once, but runs in a loop: reasoning, calling tools, observing results, and deciding what to do next until the task is complete.

Tool use / function calling - The mechanism by which a model can invoke external functions (search the web, query a database, run code) as part of its reasoning process, rather than being limited to generating text.

MCP (Model Context Protocol) - A standardized protocol for connecting models to external tools and data sources, so tool integrations aren't reinvented per-framework. Increasingly the common interface across agent frameworks and IDEs.

Orchestration - Coordinating multiple agents working together on a task - a central coordinator decomposing work and dispatching it to specialized worker agents, as opposed to one agent trying to do everything.

Handoff - A pattern where one agent, mid-task, transfers control (and relevant context) to a different, better-suited agent - common in support/triage-style systems.

Production & serving terms

Latency - Time taken to get a response. Often split into time to first token (TTFT) - how long before generation starts - and inter-token latency - how long between subsequent tokens once generation is underway. These respond to different optimizations.

Throughput - How many requests (or tokens) a system can process per unit time, as opposed to how fast any single request completes. Batching improves throughput, sometimes at the cost of individual request latency.

Continuous batching - A serving technique where new requests can join an in-progress batch as soon as a slot frees up, rather than waiting for the whole batch to finish. This is a major reason serving frameworks like vLLM outperform naive request-at-a-time loops.

Tracing - Recording the full path of a request through a system - including retries, tool calls, and fallbacks - as a structured, queryable record, rather than relying on scattered logs. Essential once an LLM app has more than one moving part.

Eval / eval loop - A systematic, often continuous process for scoring model output quality - structural checks (did the JSON parse), golden-set regression tests, or sampled judge-model scoring - used to catch quality regressions the way tests catch code regressions.

Cost & efficiency terms

Prompt / completion tokens - Most API pricing is split into input (prompt) tokens and output (completion) tokens, usually priced differently, with output tokens typically costing more per token.

Distillation - Training a smaller "student" model to mimic the behavior of a larger "teacher" model, producing something cheaper to run that retains much of the larger model's capability on the tasks it was distilled for.

Open-weight vs. closed model - Open-weight models have downloadable weights you can self-host and run under your own infrastructure (e.g. Llama, Mistral, Qwen); closed models are only accessible through a provider's hosted API (e.g. Claude, GPT). This distinction drives a real cost/control tradeoff, not just a licensing preference.

Why the vocabulary matters

None of these terms are difficult individually - the friction is that they get used loosely across blog posts, vendor docs, and papers, often without definition, on the assumption you already know them. Having a shared, precise vocabulary is what makes the difference between skimming an infra discussion and actually being able to make the tradeoff decisions it's describing - which quantization level, which attention variant, whether a task actually needs orchestration or just a well-scoped single agent.

This glossary is part of a series on AI infrastructure patterns. If a term you've run into isn't here, drop it in the comments and I'll add it.