Deep Learning and Transformers

Deep Learning and Transformers

Artificial intelligence is often introduced with phrases like: "Neural networks imitate the human brain." That analogy can be useful, but if you come from physics, mathematics, engineering, or scientific computing, there is another way to think about modern AI that may feel much more natural. A neural network is fundamentally a parameterized mathematical transformation. Training that network is an optimization problem in a very high-dimensional space. And the attention mechanism inside a Transformer can be interpreted as a learned, input-dependent interaction matrix. Once we look at AI from this perspective, much of the mystery starts to disappear. Let's build the idea from the ground up.

1. Start With an Artificial Neuron

The basic computational element of a neural network is an artificial neuron. Suppose we have several inputs: x₁, x₂, …, xₙ. Each input is associated with a weight: w₁, w₂, …, wₙ. The neuron calculates a weighted sum:

z = Σᵢ wᵢxᵢ + b

where b is called the bias. The result then passes through an activation function:

a = f(z)

In vector notation, we can write the same basic idea as:

z = w · x + b

So despite the biological name, an artificial neuron is not literally a microscopic brain cell. It is a mathematical operation. The interesting behavior begins when many of these operations are connected together.

2. Why Do We Need Activation Functions?

Suppose we create several neural-network layers but use only linear transformations. The first layer might be:

h₁ = W₁x

The second layer could be:

h₂ = W₂h₁

Substituting the first expression into the second gives:

h₂ = W₂W₁x

But W₂W₁ is simply another matrix. So no matter how many purely linear layers we stack, the entire system can still collapse into one larger linear transformation. We have added depth, but not much expressive power.

That changes when we introduce nonlinearity. Now suppose the first layer becomes:

h₁ = f(W₁x + b₁)

and the next layer becomes:

h₂ = f(W₂h₁ + b₂)

The activation function f prevents the whole network from reducing to a single linear transformation. One of the most common activation functions is ReLU:

ReLU(x) = max(0, x)

If the input is positive, ReLU keeps it. If the input is negative, ReLU returns zero. This simple nonlinearity allows networks to represent much more complicated relationships. That is one of the foundations of deep learning.

3. From Neural Networks to Deep Learning

A deep neural network contains many transformation stages. Conceptually, information moves through something like this:

x
↓
f(W₁x + b₁)
↓
f(W₂h₁ + b₂)
↓
...
↓
y

Each layer transforms the representation created by the previous layer. For an image-processing system, early layers may respond to relatively simple structures such as edges and local color changes. Later layers can combine those structures into larger patterns. Those patterns can then be combined again into increasingly useful internal representations.

The entire network can be thought of as one large parameterized function:

y = F(x; θ)

Here, θ represents all the parameters inside the model. Those parameters include weights and biases. A modern neural network can contain millions or billions of them. This gives us a useful mental model:

• A deep neural network is a very high-dimensional parameterized function.
• The architecture determines the structure of the function.
• Training determines the numerical values of its parameters.

4. Training Is an Optimization Problem

Now suppose the network makes a prediction: ŷ, while the desired answer is: y. We define a loss function:

L(ŷ, y)

The loss measures how far the prediction is from the desired result. Training then asks a very mathematical question: Which parameter values make the loss smaller?

Conceptually, we are trying to find:

θ* = arg minθ L(θ)

In other words, we want a parameter configuration that minimizes the loss. Now imagine the loss as a function of every parameter:

L(θ₁, θ₂, …, θₙ)

If the network contains a billion parameters, then this loss is defined over a billion-dimensional parameter space. We cannot visualize that space directly. But conceptually, we can still imagine a landscape containing regions of higher and lower loss.

For someone coming from physics, this is a very useful perspective: Neural-network training ≈ optimization in a huge-dimensional landscape. Instead of explicitly programming every rule the system should follow, we search for parameter values that allow the model to reproduce useful patterns in data.

5. Gradient Descent

How do we know which direction to move in this enormous parameter space? We calculate the gradient. The gradient of the loss can be written as:

∇θ L

It tells us how the loss changes when we make small changes to the parameters. A basic gradient-descent update looks like this:

θₜ₊₁ = θₜ − η∇θL

Here:

• θₜ represents the current parameters
• ∇θL represents the gradient of the loss
• η is the learning rate
• θₜ₊₁ represents the updated parameters

The learning rate controls the size of each step. If the step is too large, the optimizer may jump past useful regions. If the step is too small, training may become extremely slow.

Conceptually, gradient descent repeatedly asks: Which small change in the parameters should reduce the loss? Then it makes that change and repeats the process. Modern training algorithms are more sophisticated than basic gradient descent, but this core idea remains central.

6. What Does Backpropagation Actually Do?

A deep neural network is a composition of many functions. Conceptually:

F = fₙ ∘ fₙ₋₁ ∘ ... ∘ f₁

To train the network, we need to know how the final loss depends on parameters buried deep inside those functions. For example:

∂L / ∂Wᵢ

Backpropagation gives us an efficient way to calculate these derivatives. At its core, backpropagation is an application of the chain rule. Suppose:

y = f(g(x))

Then:

dy/dx = (df/dg)(dg/dx)

A deep neural network may contain thousands of connected mathematical operations. Backpropagation applies this principle repeatedly through the computational graph.

During the forward pass, information moves through the model and produces a prediction. The loss is calculated. Then the derivatives are propagated backward through the computation so the system can determine how changes in earlier parameters would affect that loss. Those gradients are then used by the optimizer to update the parameters.

This gives us another useful way to think about neural networks: The model is a computational graph, and backpropagation computes derivatives through that graph.

7. Then Transformers Changed the Game

Deep learning existed long before Transformers. But sequence problems such as language create a special challenge. Words do not exist independently. The meaning of one word often depends on other words that appeared earlier - sometimes much earlier - in the sequence.

Transformers introduced an especially powerful mechanism for handling these relationships: attention. Instead of forcing information to move only step-by-step through the sequence, attention allows different elements of the sequence to interact directly. The central operation is:

Attention(Q, K, V) = softmax(QKᵀ / √dₖ)V

This equation may look intimidating at first. But each part has a clear role. Let's unpack it.

8. Queries, Keys, and Values

Each token representation is transformed into three vectors:

• Q = queries
• K = keys
• V = values

A useful intuition is:

• Query - What information am I looking for?
• Key - What kind of information do I contain?
• Value - What information should I contribute if I am relevant?

The model compares queries with keys using:

QKᵀ

The result is a matrix of interaction scores. Those scores describe how strongly different elements of the sequence relate to one another. If there are N tokens, we can describe an individual score as:

Aᵢⱼ

This represents how strongly token i should attend to token j. The important point is that these relationships are calculated from the current input. They are not simply fixed in advance.

9. Attention as an Interaction Matrix

This is where the physics intuition becomes especially interesting. The operation:

QKᵀ

creates a matrix describing relationships between elements of the sequence. Conceptually, imagine something like:

              Token1  Token2  Token3  ...  TokenN
         ┌────────────────────────────────────────┐
Token1   │  a₁₁    a₁₂    a₁₃   ...   a₁ₙ        │
Token2   │  a₂₁    a₂₂    a₂₃   ...   a₂ₙ        │
Token3   │  a₃₁    a₃₂    a₃₃   ...   a₃ₙ        │
   ⋮     │  ⋮      ⋮      ⋮     ⋱     ⋮          │
TokenN   │  aₙ₁    aₙ₂    aₙ₃   ...   aₙₙ        │
         └────────────────────────────────────────┘

The raw scores are then normalized with softmax. Conceptually:

Pᵢⱼ = exp(Aᵢⱼ) / Σⱼ exp(Aᵢⱼ)

These normalized weights determine how strongly information from one token contributes to another token's updated representation. The value vectors are mixed according to those weights.

This leads to one of my favorite physics-inspired interpretations of Transformers: Attention ≈ a learned, input-dependent interaction matrix.

There is an important difference from a fixed physical interaction matrix, however. The attention matrix depends on the current input. A new sequence creates new interactions. The system is effectively asking: Which elements should interact strongly in this particular configuration?

10. A Simple Language Example

Consider this sentence:

"The animal didn't cross the street because it was tired."

What does it refer to? Most likely, the animal. The model must connect information from different positions in the sequence. Attention allows the representation associated with it to interact strongly with the representation associated with animal.

Now consider:

"The truck couldn't cross the bridge because it was broken."

This time, it most likely refers to the bridge. The token it is unchanged. But the context is different. Therefore the attention pattern can also be different.

That is one of the fundamental strengths of Transformers. The relationships among elements are not completely hard-coded. They are calculated dynamically from the current input.

11. Multi-Head Attention

Transformers usually do not calculate just one attention pattern. They calculate several attention patterns in parallel. This is called multi-head attention. An individual attention head can be written conceptually as:

headᵢ = Attention(Qᵢ, Kᵢ, Vᵢ)

Several heads are then combined:

MultiHead(Q, K, V) = Concat(head₁, head₂, …, headₕ) Wᴼ

Different attention heads can capture different relationships. One may become useful for relatively local structure. Another may capture longer-range dependencies. Another may respond to different semantic or structural patterns.

But we should be careful not to assume that every attention head always has one neat, human-readable job. Neural-network representations are often distributed across many components. Still, multi-head attention gives the Transformer multiple interaction channels through which information can flow.

12. A Transformer Is More Than Attention

Attention is central to the Transformer architecture. But attention alone is not the entire Transformer. A simplified Transformer block looks roughly like this:

Input Representations
       ↓
   Self-Attention
       ↓
 Feed-Forward Network
       ↓
  Next Transformer Layer

Modern Transformer blocks also use important components such as residual connections and normalization. A slightly more realistic conceptual picture looks like:

Input
       ↓
   Self-Attention
       ↓
 Residual Connection + Normalization
       ↓
 Feed-Forward Network
       ↓
 Residual Connection + Normalization
       ↓
      Output

This process is repeated across many layers. We can imagine the internal representations evolving like this:

X⁽⁰⁾ → X⁽¹⁾ → X⁽²⁾ → ... → X⁽ᴸ⁾

At each stage, the representation of each token can change. Information from other tokens can influence it through attention. The feed-forward network then performs additional nonlinear transformations. Layer after layer, the model builds increasingly rich representations of the input.

13. How Does This Become a Large Language Model?

A language model begins with tokens: t₁, t₂, …, tₙ. Each token is mapped into a numerical representation. Those representations pass through many Transformer layers. Eventually, the model produces numerical scores for possible next tokens. Those scores are converted into a probability distribution.

Conceptually:

P(tₙ₊₁ | t₁, t₂, …, tₙ)

For example, the model might produce something like:

• P("physics") = 0.35
• P("science") = 0.21
• P("experiment") = 0.08

A decoding strategy then chooses the next token. That token becomes part of the context. Then the process happens again:

t₁, t₂, …, tₙ → tₙ₊₁ → tₙ₊₂ → ...

At its core, a language model repeatedly predicts what token is likely to come next given the context. That may sound surprisingly simple. But when this objective is scaled across enormous datasets, large models, and powerful computing infrastructure, remarkably sophisticated behavior can emerge.

14. LLMs Are Not Giant Databases

A common misconception is that a large language model is simply an enormous database containing billions of stored sentences. That is not the best way to think about it. The model learns statistical structure through its parameters.

Conceptually:

P(next token | context; θ)

The parameter set θ contains the numerical structure learned during training. Knowledge is distributed through these parameters rather than being stored as a clean collection of sentences waiting to be retrieved.

When you provide a prompt, the model performs inference. The text is represented as tokens. Those tokens become numerical vectors. The Transformer repeatedly transforms those vectors. Attention allows information to flow between relevant parts of the context. Layer after layer modifies the internal representations. Finally, the model produces a probability distribution over possible next tokens.

So an LLM is better understood as a huge nonlinear transformation than as a conventional lookup database.

15. A Physicist's Mental Model of Modern AI

Now the pieces fit together. An artificial neuron performs a simple transformation:

a = f(w · x + b)

Many artificial neurons form a neural network. Many layers give us deep learning. Training is optimization in a high-dimensional space. Backpropagation efficiently computes gradients through the computational graph. Transformers introduce attention - a learned, input-dependent interaction matrix - allowing elements of a sequence to relate to one another dynamically. Stacked into many layers with feed-forward transformations, these mechanisms become large language models that predict the next token at each step.

From this perspective, modern AI is not magic. It is a very large parameterized mathematical transformation, trained by optimization, structured by attention, and scaled until complex behavior emerges.