What Happens Inside an LLM Before It Generates Your First Token?

Every day, billions of people ask AI assistants questions. The answers appear almost instantly, making the process feel effortless. Yet before the very first word appears, one of the most sophisticated inference pipelines ever built has already executed billions of mathematical operations. This invisible journey is where modern AI truly comes alive. From tokenization and embeddings to GPU inference, KV Cache, FlashAttention, and speculative decoding. Part 1 - From Human Language to Machine Understanding The first word an AI generates is actually the last step of a remarkably complex journey. When you open ChatGPT, Claude, Gemini, or any modern Large Language Model, the interaction feels almost magical. You type a question. You press Enter. A brief pause follows. Then, words begin appearing one after another-as if the model is thinking in real time. To most users, it feels like the AI is simply "reading" the prompt and immediately responding. But that's far from what actually happens. Between the moment you press Enter and the moment the very first token appears on your screen, an extraordinary pipeline unfolds inside the model. That single pause hides millions-sometimes billions-of mathematical operations happening across GPUs. The model isn't reading English. It isn't reasoning with words. It isn't storing paragraphs inside its memory like humans do. Instead, it transforms language into mathematics. Only after completing an intricate sequence of computations does it predict the first token. This article is not another simplified explanation of "how ChatGPT works." Instead, we'll walk through the same inference pipeline that powers production-grade Large Language Models used by companies like OpenAI, Anthropic, Google DeepMind, Meta, and Mistral. By the end of this series, you'll understand not only what happens inside an LLM-but why every step exists. The Journey Before the First Token Before diving into individual components, it's helpful to visualize the complete pipeline. flowchart LR A["👤 User Prompt"] -->B["Tokenizer"] B-->C["Token IDs"] C-->D["Embeddings"] D-->E["Positional Encoding"] E-->F["Transformer Layers"] F-->G["Probability Distribution"] G-->H["Sampling"] H-->I["First Generated Token"] Although this diagram appears simple, every block represents an entire field of research. Some stages execute only once. Others repeat for every generated token. In this first part, we'll focus on the earliest stages-the ones responsible for converting human language into something a neural network can understand. Why LLMs Don't Understand Words One of the biggest misconceptions about AI is that models understand language the way humans do. They don't. Humans process meaning. Machines process numbers. Imagine asking ChatGPT: Explain recursion using a simple analogy. You see a sentence. The model does not. Before anything else happens, your sentence must become numerical data. Because neural networks cannot perform calculations on letters. They only understand vectors, matrices, and tensors. This transformation is the foundation of modern Natural Language Processing. Without it, GPT, Claude, Gemini, and Llama simply cannot operate. Step 1 - Receiving the Prompt Let's use a simple prompt throughout this article: Why is the sky blue? From the user's perspective, this looks like plain English. Inside the inference server, however, the prompt arrives as raw Unicode text. At this stage, the model hasn't processed anything. No intelligence has been applied. No prediction has been made. The inference server simply receives a sequence of characters. Conceptually, it looks like this: | Human View | Machine View | |----------------------|------------------------| | Why is the sky blue? | Raw Unicode characters | This is still unusable for the neural network. The next stage changes everything. Step 2 - Tokenization Tokenization is often described as "splitting text into words." That explanation is convenient. It's also inaccurate. Modern LLMs rarely tokenize by words. Instead, they use subword tokenization, allowing them to efficiently represent nearly every language, programming syntax, emoji, and even spelling mistakes. Consider this sentence: Artificial Intelligence A tokenizer might split it like this: | Text | Token | |------|------------------| | Artificial | Art | | Artificial | ificial | | Intelligence | Intel | | Intelligence | ligence | Different models use different vocabularies. GPT, Claude, Gemini, and Llama all have their own tokenizer implementations. The exact tokens differ. The underlying principle remains the same. Instead of understanding words, the model understands predefined pieces of text. Why Not Store Every Word? Imagine storing every possible English word. Now add: - Hindi - Japanese - Chinese - Python code - JavaScript - Emojis - URLs - Mathematical equations - Misspellings - Company names - Future slang The vocabulary would become impossibly large. Subword tokenization solves this elegantly. A small vocabulary can represent virtually unlimited text combinations. That's one reason modern LLMs scale so effectively. Real Example Suppose you type: Unbelievable A tokenizer might produce something similar to: | Token | |--------| | Un | | believe | | able | Instead of memorizing the entire word, the model builds meaning from reusable pieces. This dramatically reduces vocabulary size while increasing flexibility. Tokens Are Not Words This distinction is surprisingly important. Consider these examples. | Input | Approximate Tokens | |----------------|------------------| | Hello | 1 | | Good morning | 2-3 | | Artificial Intelligence | 3-5 | | 👋 | 1-3 | | console.log() | Multiple | This is why API pricing is based on tokens, not words. A thousand words can produce significantly more-or fewer-than a thousand tokens. Understanding this difference becomes essential when optimizing AI applications for latency and cost. We'll revisit token economics later in this series. Engineering Insight The tokenizer is not part of the neural network. This surprises many developers. The tokenizer is simply a preprocessing component. It converts text into token IDs before the Transformer begins inference. Only after tokenization does the actual LLM start working. Step 3 - Converting Tokens into IDs Tokens themselves are still text fragments. The neural network cannot process strings either. Each token is therefore mapped to an integer. Example: | Token | Token ID | |---------|----------| | Why | 4921 | | is | 318 | | the | 262 | | sky | 6766 | | blue | 4171 | | ? | 30 | These numbers have no mathematical meaning by themselves. They simply act as unique identifiers inside the model's vocabulary. Think of them like dictionary indexes. The model still hasn't begun "thinking." It has only converted language into references. Why IDs Alone Are Meaningless Suppose someone tells you: 4921 318 262 6766 4171 30 Could you infer that this means: Why is the sky blue? Of course not. These numbers contain no semantic information. They merely point to entries in a vocabulary table. Meaning enters the system in the next stage. Step 4 - Embeddings: Where Language Becomes Mathematics This is where the magic truly begins. Each token ID is transformed into a high-dimensional vector called an embedding. Instead of representing "blue" as: 4171 the model converts it into something conceptually like: [-0.18, 1.42, -0.77, 0.56, ...] Not four numbers. Not forty. Modern LLM embeddings often contain thousands of dimensions. Each dimension captures subtle statistical relationships learned during training. Words with similar meanings naturally occupy nearby regions in this mathematical space. For example: King Queen Prince Princess end up close together. Similarly, Python Java JavaScript C++ form another neighborhood. The model doesn't understand these concepts linguistically. It understands them geometrically. Meaning becomes distance. Similarity becomes direction. Language becomes linear algebra. Visualizing Embedding Space Imagine a simplified two-dimensional world. Animal 🐶 🐺 🦊 -------------------------------------------- 🚗 🚀 Technology Real embeddings don't exist in two dimensions. They exist in hundreds or thousands. But the intuition remains valid. Concepts that frequently appear in similar contexts become neighbors inside vector space. Why Embeddings Changed AI Forever Before embeddings, NLP relied heavily on handcrafted rules and sparse representations. Embeddings introduced something revolutionary. Instead of explicitly defining relationships, models learned them automatically from enormous amounts of text. The model isn't told that: Doctor and Physician are similar. It discovers this statistically. That capability transformed Natural Language Processing. Embeddings became the foundation upon which modern Transformers were built. Without embeddings, today's LLMs simply wouldn't exist. Engineering Note An embedding is not a definition. It is a learned numerical representation whose position reflects how language behaves across billions of examples. Where We Stand So Far At this point, the model has still not generated a single token. Yet it has already completed several critical stages. | Stage | Completed | |--------|--------------------| | Prompt received | ✅ | | Tokenization | ✅ | | Token IDs created | ✅ | | Embeddings generated | ✅ | | First token predicted | ❌ | Everything so far has been preparation. The real computation is about to begin. What's Coming Next So far, we've transformed human language into mathematical vectors. But vectors alone don't create intelligence. The model still has no understanding of context. It doesn't know which words relate to each other. It doesn't know what part of the sentence is important. It doesn't know whether "bank" refers to a financial institution or the side of a river. That understanding emerges inside the Transformer-the architecture that revolutionized artificial intelligence. In Part 2, we'll step