Why Transformers Need Positional Encoding For Time Series: A Visual Guide

Why Transformers Need Positional Encoding For Time Series: A Visual Guide From scalar observations to self-attention, and how positional information restores sequence order While digging into foundation models for time series, I realized that I could not really understand them without first understanding transformers. I did not want to use these models as black boxes, so I started tracing the ideas backward, from foundation models to transformers, and from transformers to self-attention. What made the transition interesting is that although transformers were originally built for language, the core idea carries naturally to time series. The two modalities are very different, but they share something fundamental: both are sequences, and in both cases, order changes meaning. In language, dog bites man is very different from man bites dog. Time series are no different. A temperature of yesterday and today tells a different story from yesterday and today. The values may be the same, but their order changes the meaning of the sequence. The question is that if self-attention looks at all observations at once, how does a transformer know which observation came first, which came later, or how far apart two observations are? That question led me to positional encoding. What surprised me most was how such a simple mathematical idea could give a Transformer a sense of order. The exact techniques have evolved considerably since then, but the underlying problem remains the same. This article is my attempt to build that intuition from the ground up, starting with a simple time series and following the path from raw observations to self-attention and finally to positional encoding. From scalar observations to vector representations Consider a simple time series containing the temperature recorded over five weekdays: Each observation is only a scalar. A transformer, however, operates on vectors of dimensionality . The scalar observations therefore need to be mapped into that representation space first. A simple way to do this is through a learned linear projection i.e. embedding: giving us a sequence of vector representations: An embedding is a deep, abstract representation of the series in the form of a multidimensional numerical vector that encodes its features and that the model understands. [1] Each captures information about the observed value at that timestep, but at this point, it is just a representation of the observation. The important word here is learned. The model is not given a predefined vector representation for a temperature such as . The parameters and are learned during training so that the resulting representations become useful for the task. At this point, represents what was observed. It does not yet tell the model where that observation occurred in the sequence. How self-attention builds context? Self-attention allows each observation to use information from the rest of the sequence. Suppose we want to update Friday’s representation. The model first creates three learned projections from every : The matrices , and are also learned during training. The model is not told beforehand what a useful query, key, or value should look like. For Friday, its query is compared with the keys of all observations: . Each comparison produces an attention score: which measures how relevant observation 'j' is when updating Friday’s representation. The scaling factor prevents the dot products from growing too large as the dimensionality of the query and key vectors increases. These scores are passed through a softmax function to convert them into attention weights: Finally, those weights are used to combine the value vectors: So is Friday’s representation before incorporating information from the rest of the sequence, while is its context-aware representation after self-attention. In short: Queries and keys learn which observations are relevant to one another. Values carry the information that is combined to form the new representation. What happens if we shuffle the sequence? Now comes the important question. Suppose the same five temperature observations are rearranged. The values themselves have not changed; only the order has. After the learned projection, we still have the same set of value representations, just rearranged. Self-attention can still compare each representation with all the others. The same query, key, and value projections are applied, and the same kinds of pairwise relationships can still be computed. What has disappeared is the temporal structure. Nothing inside says that it originally came from Thursday. Nothing inside says that it occurred after . Also, if Wednesday and Friday have the same temperature value, the learned projection will map them to the same embedding vector. Without positional information, the model therefore has no way to distinguish which embedding came from Wednesday and which came from Friday. This is the key limitation: Self-attention can learn which observations are related, but without an additional positional signal, it has no built-in way to know where those observations occurred in the sequence. What should positional information tell the model? If self-attention does not know the order of the observations, then the next question is: what kind of positional information would be useful? At a minimum, we would want the model to know: Which position an observation belongs to? Position 2 should be distinguishable from position 20.Which observation came before or after another? The model should be able to distinguish from .How far apart are two observations? In time series, the difference between , and can be important.That nearby positions are related in a structured way. Position 10 and position 11 should not look like two completely unrelated identifiers.That the representation remains useful over longer sequences. Ideally, the positional scheme should still provide meaningful structure as the sequence grows. For time series, the third property is especially useful. A model may care about an observation one step ago because of short-term dependence, or seven steps ago because of a weekly seasonal pattern. So positional information should do more than simply assign a unique label to each timestep. It should give the model a structured representation of order and relative distance. How can we represent position? We now know what information is missing. The next question is how to represent it. A simple way to represent position would be to assign each timestep a number: But feeding the raw position directly into the model is not ideal. The values keep growing with sequence length, and a single number does not give the model a rich representation of positional relationships. One of the original Transformer’s solutions was sinusoidal positional encoding, where each position is represented using sine and cosine functions at different frequencies. Why sine and cosine? Start with the simplest two-dimensional example: As changes, the positional vector moves smoothly around a circle that allows nearby positions to have different but still related representations. More importantly, moving forward by the same number of steps produces the same kind of change in the positional representation. For example, an offset of 7 positions has the same mathematical relationship whether we move from position 3 to 10 or from position 20 to 27. That is useful for time series because relative distance often matters: can represent very different temporal relationships. The full sinusoidal positional encoding extends this idea across many dimensions: Different dimensions use different frequencies. Some change quickly across nearby positions, while others change much more slowly. One useful way to think about this is as many clocks running at different speeds. Together, their readings give every position a structured positional signature. So instead of giving timestep (t) only a number, we give it a vector: that contains information about its position and its relationship to other positions. Combining value and position After adding positional information, each timestep is represented as: where represents the observed value and represents its position in the sequence. Self-attention now builds its queries and keys from this combined representation: So when the model computes an attention score, the comparison is no longer based only on the observed values. The query and key vectors were created from representations that already contain positional information. As a result, the model can learn relationships that depend on both: what was observed, and where the observations occurred in the sequence. For a time series, this means the model can potentially learn that an observation one timestep ago should be treated differently from one seven timesteps ago, even if their values are similar. Positional encoding therefore does not tell the model explicitly which lags are important. It gives self-attention the information needed to learn which positional relationships matter for the task. Summary We started with a simple problem: self-attention can compare every observation with every other observation, but by itself it does not know the order in which those observations occurred. The solution is to enrich each value representation with a positional representation : Self-attention then builds its queries, keys, and values from rather than from the value representation alone. This means the attention mechanism can learn relationships based not only on what was observed, but also on where that observation occurred in the sequence. For regularly sampled time series, this makes relationships such as t−1, t−7, or t−24 accessible to the model. A lag of one step may capture short-term dependence, while a lag of seven or twenty-four steps may correspond to a seasonal pattern. Positional encoding, however, represents sequence position, not necessarily real-world time. If observations are irregularly spaced, being one p