Embeddings: Meaning as Numbers

When you ask a chatbot for a pasta recipe, it doesn’t understand “pasta” the way you do. It sees a list of numbers. Those numbers are not random. They are arranged so that “pasta” sits close to “spaghetti” and far from “car.” This is an embedding. It turns meaning into coordinates that a machine can measure, compare, and search. Every time a chatbot finds a relevant answer, recommends a product, or remembers a fact from earlier in the conversation, embeddings are doing the heavy lifting behind the scenes. Here is the surprising part. The machine never learns what “pasta” means. It only learns which words appear in the same kinds of sentences. That statistical shadow turns out to be so rich that it captures everything from synonyms to analogies. The rest of this article unpacks how that happens, step by step, using the same chatbot interactions you already know from this series. What exactly is an embedding? An embedding is a list of numbers that represents something discrete, a word, a token, a user ID, a product. Each number is a coordinate in a high-dimensional space. If you pick the right coordinates, similar things end up near each other. That is the entire idea. Think of a map. A city’s latitude and longitude don’t tell you its name or history. But if you know that Paris is at (48.9, 2.3) and Lyon is at (45.8, 4.8), you can measure the distance and see they are both in France. Embeddings work the same way, except they use hundreds of dimensions instead of two. Every dimension captures some latent feature of the input, learned from data. The model never labels those features. It just arranges points so that words that behave similarly in text end up with similar coordinates. In a chatbot, every token from Part 2 gets its own embedding vector. The model then uses those vectors as the starting point for everything else, including the attention mechanism from Part 3. The quality of the embeddings directly determines how well the model can tell that “I need a quick dinner idea” and “fast evening meal suggestions” mean roughly the same thing. How does a chatbot turn words into numbers? You already know that the chatbot breaks your message into tokens. For the query “give me a pasta recipe,” the tokenizer might produce token IDs like [123, 45, 678, 901] . Those integers are just labels. They carry no meaning on their own. The model has a large table called an embedding matrix. It has one row for every token in the vocabulary. Each row is a vector of, say, 768 numbers. When the model sees token ID 678, it looks up row 678 and pulls out that vector. This lookup is the embedding layer. It turns a sequence of token IDs into a sequence of dense vectors that the rest of the network can process. graph TD A["Input text 'pasta recipe'"] --> B["Tokenizer produces IDs"] B --> C["Embedding matrix lookup"] C --> D["Vector for token 'pasta'"] C --> E["Vector for token 'recipe'"] This lookup is fast and differentiable. During training, the model adjusts the numbers in those rows so that the vectors become useful for the task at hand. If the model is trained to predict the next word, then words that lead to similar next-word predictions will gradually drift toward each other in the vector space. The embedding layer itself has no built-in notion of meaning. It is just a giant spreadsheet of numbers that gets updated by backpropagation. The same mechanism applies to any discrete input. When a recommendation system sees your user ID, it fetches a vector that represents your preferences. When a search engine indexes a document, it stores an embedding vector for the whole document. The table lookup is the universal first step for turning symbols into numbers a neural network can digest. Why do similar words end up with similar vectors? The key insight is the distributional hypothesis. Words that appear in similar contexts tend to have similar meanings. If you see the word “pasta” surrounded by “sauce,” “boil,” “dinner,” and “recipe,” and you see “spaghetti” in the exact same kinds of sentences, then a model that learns from those contexts will place “pasta” and “spaghetti” close together. It doesn’t know what either word means. It only knows they are interchangeable in many sentences. Early embedding methods made this explicit. They would scan a huge corpus and count how often every pair of words appeared near each other. That produced a giant co-occurrence matrix. Then they would squash that matrix down to a small number of dimensions using a technique like singular value decomposition. The result was a compact vector for each word that preserved the most important co-occurrence patterns. This is like taking a huge spreadsheet of word relationships and compressing it into a few columns that capture the gist. Modern models do something similar, but they learn the vectors on the fly while training a neural network. They don’t count everything first. Instead, they look at a small window of words, try to predict a target word from its neighbors (or vice versa), and adjust the vectors to get better at that prediction. Over millions of examples, the vectors settle into a geometry where words that predict the same contexts cluster together. This is why embeddings can solve analogies. The classic example is “king minus man plus woman equals queen.” The vectors don’t store royal titles. They store the directions that separate gender and royalty, learned from thousands of sentences where these words appear in parallel roles. How does the model learn these vectors? The most famous example is Word2Vec’s skip-gram model with negative sampling. The idea is simple. Take a sentence like “I need a quick pasta recipe.” Slide a window over it. For each center word, like “pasta,” pick a nearby context word, like “recipe.” The model’s job is to decide whether this pair is real or fake. The model computes a dot product between the embedding of “pasta” and the embedding of “recipe.” A large positive dot product means the model thinks they belong together. A negative or small dot product means they don’t. The model is also shown fake pairs, like “pasta” and “elephant,” sampled at random. It must learn to give high scores to real pairs and low scores to fake ones. { "type": "bar", "title": "Dot product similarity between word embeddings", "caption": "Illustrative similarity scores. High scores indicate words that often appear together.", "data": [ { "label": "pasta - recipe", "value": 0.8 }, { "label": "pasta - sauce", "value": 0.75 }, { "label": "pasta - car", "value": 0.1 }, { "label": "pasta - quantum", "value": -0.2 } ] } Over time, the embeddings shift so that words that often appear together get vectors with large dot products. Words that never appear together get vectors that point in different directions. Because the model sees millions of real and fake pairs, it ends up encoding subtle semantic relationships. “Pasta” and “spaghetti” both appear with “recipe,” so they both get pushed toward the same region of the space. Researchers later proved that this process is equivalent to factorizing a matrix of pointwise mutual information (PMI) between words and contexts. PMI measures how much more often two words co-occur than you would expect by chance. The model is essentially learning a low-dimensional approximation of that PMI matrix. That is why the numbers capture meaning. They are a compressed representation of statistical association. In practice, the model never builds the full PMI matrix. It learns the vectors directly from the data stream, which scales to billions of words. The embedding dimension, typically a few hundred, is a hyperparameter that controls how much information can be packed into each vector. More dimensions can capture finer distinctions, but they also cost more memory and computation. How are embeddings used beyond single words? A chatbot needs to understand whole sentences, not just individual words. After the token embeddings are looked up, the transformer layers from Part 3 mix them together using attention. The final output is a sequence of vectors, one per token, each now carrying information about the entire sentence. To get a single vector for the whole sentence, a common trick is to take the vector corresponding to a special token, like [CLS] , or to average all token vectors. This sentence embedding can then be compared to other sentence embeddings using cosine similarity. If two sentences have a high cosine similarity, the model considers them semantically close. This is how a chatbot knows that “how do I make pasta” and “pasta cooking instructions” are asking the same thing. The query is turned into a sentence embedding, and the chatbot compares it to embeddings of candidate answers or documents. The one with the highest similarity is returned. Modern sentence embedding models are often trained with contrastive learning. They take a sentence, create two slightly different versions (like by dropping words or using a paraphrase), and force the model to make those two versions have nearly identical embeddings. At the same time, they push embeddings of unrelated sentences apart. This makes the embedding space more uniform and reliable for similarity comparisons. This same idea powers recommendation systems. A user’s interaction history can be averaged into a user embedding. Items get their own embeddings. The dot product between user and item embeddings predicts how much the user will like the item. Behind every “you might also like” is a nearest-neighbor search in embedding space. What does this mean for search, recommendations, and memory? When you type a query into a chatbot, it doesn’t do a keyword match. It converts your query into an embedding and finds the stored passages whose embeddings are closest. This is semantic search. It works even when the words don’t overlap at all. “Inexpensive lodging near the beach” and “budget hotel by the sea” will map to nearby points, so the system retrieves the same results. Th