The Transformer: How GPT Learns to Pay Attention

In Part 1, I focused on understanding how a neural network learns. I built neurons, experimented with weights and biases, implemented gradient descent, and worked through backpropagation. I also learned how easily a model can go wrong when gradients or training parameters aren't handled correctly.

But there was still one question I hadn't answered: How do we get a neural network to work with language? A model that learns to add two numbers is one thing. A model that reads a sentence, considers the surrounding words and predicts what should come next is a completely different challenge.

I started exploring the architecture that made modern language models possible: the Transformer. And before I could understand its attention mechanism, I had to understand how text becomes something a neural network can process.

1. Computers don't understand text the way we do

Consider this sentence: The developer is building an AI model. We see words, grammar and meaning. A neural network doesn't receive that sentence as a human-readable string and magically understand it. It needs a numerical representation. That is where tokenisation comes in.

A tokeniser splits text into smaller units called tokens. Depending on the tokeniser, a token can be a character, a word, part of a word, or another text unit.

For our first Shakespeare GPT implementation, I chose character-level tokenisation. Why characters? Because it made the first language-model implementation easier to understand. We could build a small vocabulary, assign an ID to each character, and focus on learning the Transformer rather than starting with a complicated subword tokeniser.

Imagine this text: hello. A simplified character vocabulary might look like:

const vocabulary = ["e", "h", "l", "o"];

Each character gets an ID:

• e → 0
• h → 1
• l → 2
• o → 3

The sentence becomes: [1, 0, 2, 2, 3]

In our implementation, we built functions to:

• Read the text.
• Build a vocabulary of unique characters.
• Create a character-to-ID mapping.
• Encode text into IDs.
• Decode IDs back into text.

The model doesn't need to know that "1" represents the letter "h". It needs a consistent mapping so that the same character always receives the same ID.

This is a simple implementation, but it introduced an important principle: A language model operates on token IDs, not directly on the original text. A character-level tokeniser maps each unique character to an ID, so "hello" becomes [1, 0, 2, 2, 3].

2. Token IDs aren't enough: enter embeddings

Now suppose the tokeniser gives us: [1, 0, 2, 2, 3]

These are just identifiers. The difference between token ID 1 and token ID 2 isn't necessarily meaningful. Token IDs are not semantic measurements. We need to turn them into vectors. That's the job of an embedding layer.

Imagine a vocabulary of 65 characters and an embedding dimension of 32. Our embedding matrix has the shape: W E ​ ∈ R 65 × 32

That means:

• There are 65 rows, one for every character.
• Each row contains 32 learned numbers.

When the model receives token ID 1, it looks up row 1 of this matrix. In JavaScript, the basic idea looks like this:

function getEmbedding(tokenId, embeddingMatrix) {
  return embeddingMatrix[tokenId];
}

In our actual implementation, we built an Embedding layer with a trainable weight matrix and a backward function that accumulates gradients into the rows used by the input tokens. The embedding values start with an initialisation strategy. During training, they change as the model learns.

This was an important distinction for me: The tokeniser assigns IDs. The embedding layer learns numerical representations for those IDs. They're not the same thing. The embedding layer is a learned lookup table: token ID 1 selects row 1 of the 65 × 32 matrix, a vector of 32 learned numbers.

3. A language model needs to know where tokens appear

Consider:

• The dog chased the cat.
• The cat chased the dog.

The same words appear in both sentences, but their order changes the meaning. If we only gave the model token embeddings, we would still need to provide information about the positions of those tokens. That's why Transformers use positional information.

For our implementation, we built a positional embedding layer. For example, if the maximum sequence length is 64 and the embedding dimension is 32, the positional embedding matrix has shape: W P ​ ∈ R 64 × 32

Position 0 gets one learned vector, position 1 gets another, and so on. The token embedding and positional embedding are added: X = E + P

Where:

• E is the token embedding.
• P is the positional embedding.
• X is the combined representation passed into the Transformer.

This is how the model gets both token identity and position information. Each position's input is its token embedding plus its positional embedding: X = E + P.

4. Self-attention: the concept that changed everything

Now we get to the part I really wanted to understand. Imagine reading: The developer opened the laptop because it was slow. What does "it" refer to? A reader uses the surrounding context to understand the sentence.

Self-attention gives a Transformer a mechanism to calculate relationships between tokens in a sequence. Instead of processing every token independently, it allows a token's representation to be influenced by other relevant tokens.

The mechanism uses three learned projections:

• Query (Q)
• Key (K)
• Value (V)

I remember understanding that the model calculates these three things, but initially struggling with the more important question: Why do we need three different representations of the same input? Here's the intuition I eventually found useful.

Think of a Query as what a token is looking for. A Key describes what another token offers as a match. A Value contains the information that can be passed along if that token receives attention. This is an analogy, not a literal conversation happening inside the model. The actual mechanism is matrix multiplication and weighted aggregation.

How are Q, K and V created? They come from three trainable weight matrices: Q = X W Q ​ K = X W K ​ V = X W V ​

The input representation X is transformed three different ways. The matrices W Q ​ , W K ​ , and W V ​ are learned parameters. They are typically initialised before training and updated through backpropagation, just like other trainable weights.

In our JavaScript implementation, I created separate Linear layers for the three projections. Conceptually:

const q = queryProjection.forward(x);
const k = keyProjection.forward(x);
const v = valueProjection.forward(x);

All three receive the same input, but they have different learned weights. That distinction matters. The model isn't manually told what each Query, Key or Value should contain. Training shapes those projections. Q, K and V come from the same input X, passed through three separate Linear layers with their own learned weights.

5. How does attention actually work?

Once we have Q, K and V, we need to calculate attention scores. The basic scaled dot-product attention equation is: Attention ( Q , K , V ) = softmax ( d k ​ ​ Q K T ​ ) V

Let's break it down.

Step 1: Compare Queries with Keys Q K T. The dot product measures how strongly a Query matches a Key. For a sequence with multiple tokens, this produces a matrix of attention scores. Each score represents how strongly one position matches another position according to the learned projections.

Step 2: Scale the scores d k ​ ​ Q K T ​. Why divide by the square root of the key dimension? As the dimension increases, dot products can grow in magnitude. Large scores can make softmax distributions extremely concentrated, which can lead to small gradients. Scaling helps keep the scores in a more manageable range.

Step 3: Apply softmax. Softmax converts scores into probabilities that sum to 1 across the relevant positions. For example:

• Scores: [2.0, 1.0, 0.5]
• Probabilities: [0.63, 0.23, 0.14]

The exact values depend on the scores, but the important idea is that the model now has normalised attention weights.

Step 4: Combine the Values AttentionWeights × V. Each Value is multiplied by its corresponding attention weight, and the weighted values are combined. The output is a new representation containing information from the positions that received attention.

That's the essential mechanism. Scaled dot-product attention: compare Queries with Keys, scale by √dₖ, apply softmax, then take a weighted sum of the Values. The model isn't searching a dictionary for the correct answer. It's learning transformations and relationships that help construct useful representations of the context.

6. Causal masking: why GPT cannot look into the future

There is one more important detail. We're building a GPT-style language model that predicts the next token. When predicting the next character in a sequence, the model must not use characters that come after the position it's predicting. Otherwise, it would have access to the answer during training. That's why we use causal masking.

Imagine this sequence: T H E. When processing the first position, the model can only attend to the first position. At the second position, it can attend to the first and second. At the third position, it can attend to all three.

The attention mask blocks future positions. Conceptually, the mask looks like this:

100
110
111

The exact implementation uses a large negative value for blocked attention scores before applying softmax, so their resulting probabilities are effectively zero. This is called causal or autoregressive attention. It is fundamental to next-token generation. The causal mask lets each position attend only to itself and earlier positions. Blocked scores get a large negative value, so softmax makes them effectively zero.

7. Why multiple attention heads?

One attention head calculates one set of attention relationships. But language has many kinds of relationships. A token might need information about nearby words, a subject, an earlier name, or a broader part of the sentence.

Multi-head attention lets the model calculate several attention transformations in parallel. Each head has its own Q, K and V projections. Their outputs are concatenated and passed through another learned projection.

For example, if the model dimension is 32 and we use two heads of dimension 16: 2 × 16 = 32. The concatenated output has 32 features, matching the model dimension. Two heads of dimension 16, each with its own Q, K and V, are concatenated to 32 features and passed through an output projection.

In our implementation, we built individual attention heads, combined their outputs, and used an output projection to bring the result back to the model dimension. The important thing is that each head has separate learned parameters. They can learn different attention patterns, although we shouldn't assume in advance that a particular head will learn a specific linguistic role.

8. The Transformer block: putting the pieces together

Attention is powerful, but it isn't the entire Transformer. A Transformer block also contains a feed-forward network, normalisation and residual connections. Our implementation used a Pre-LayerNorm structure. Conceptually:

• Input
• LayerNorm
• Multi-Head Attention
• Residual Connection
• LayerNorm
• Feed-Forward Network
• Residual Connection
• Output

Feed-forward network. The feed-forward network applies learned transformations to each token position. Our implementation used: Linear → GELU → Linear. It expands the feature dimension, applies a nonlinearity, and projects back to the original model dimension.

Residual connections. A residual connection adds the input of a sub-layer to its output. Output = X + F ( X ). Why? It provides a direct path for information and gradients through the network, helping deeper models train more effectively.

LayerNorm. LayerNorm normalises the feature values for each token position. It also has trainable parameters:

• Gamma: scale.
• Beta: shift.

These parameters are learned during training. I spent time understanding how the gradients for gamma and beta are calculated and why updating them affects the normalisation behaviour across the model.

It was another reminder that a Transformer isn't just attention. Its surrounding components are important too. A Pre-LayerNorm Transformer block: LayerNorm, multi-head attention and a residual connection, then LayerNorm, the feed-forward network and another residual connection.

9. From one Transformer block to GPT

A GPT model stacks Transformer blocks. The overall structure looks like this:

• Token IDs
• Token Embeddings + Positional Embeddings
• Transformer Block 1
• Transformer Block 2
• ...
• Final LayerNorm
• Language Model Head
• Next-token Logits

The language model head projects the final hidden representation into vocabulary-sized scores. If our vocabulary contains 65 characters, the output at every position contains 65 logits. These logits represent the model's unnormalised scores for the possible next characters. A softmax can turn them into probabilities.

The full GPT stack: token IDs, token + positional embeddings, stacked Transformer blocks, a final LayerNorm and the language model head, which gives 65 logits per position.

For next-token training, we compare those predictions against the actual next character in the dataset. And that brings us back to the learning process from Part 1. The Transformer produces predictions. The loss measures how good they are. Backpropagation calculates gradients. The optimiser updates the weights. The same basic learning loop now operates over a much more sophisticated architecture.

10. What I learned while implementing the architecture

One thing I want to emphasise is that reading the Transformer equations and implementing a Transformer are two very different experiences. When you read: softmax ( d k ​ ​ Q K T ​ ) V it looks compact.

But implementing it means handling:

• Batch dimensions.
• Sequence dimensions.
• Feature dimensions.
• Matrix transposes.
• Causal masks.
• Softmax stability.
• Backpropagatio