Learning neural networks through mental models

For a while, I understood neural networks mostly mechanically. Data entered the network, passed through layers, and eventually produced a prediction. Loss was calculated, gradients computed, and weights updated. I could follow the sequence but the pieces still felt strangely arbitrary. Why this structure? Why did stacking layers make a network so powerful? Why were activation functions so important? I understood much of what the network was doing, step by step, without feeling like I understood the network as a whole.

Then I watched Josh Starmer's StatQuest video on the key ideas underlying neural networks. He presented neural networks as systems that sculpt functions. Instead of following data through layers, I could zoom out and picture what those transformations were collectively constructing. That shift helped immediately. Then I got greedy and started looking for other ways to think about neural networks. By the time I was done, I realised that different mental models made different questions easier to answer. Some let me zoom out and collapse the network into one idea. Others did the opposite, breaking it into smaller transformations or computations. Some connected neural networks to ideas I already understood; others made me reinterpret things I thought I understood already. Together, they made neural networks feel much less mysterious. Learning to switch between these mental models was the breakthrough I needed.

Neural networks as stacks of transformations

The mechanical understanding I started with was essentially viewing neural networks as a stack of transformations. Each layer progressively transforms the data before passing it on, where x is the input, h' are intermediate hidden representations, and y is the final output.

This was probably my default mental model before I knew to call it one. It made the forward pass easy to follow: the input is transformed step by step until the network produces an output. This perspective helps explain why neural networks have depth at all. Each layer can transform the output of the previous layer, allowing the network to build increasingly complex computations. Earlier layers produce simpler transformations that later layers can refine further. At the same time, this perspective kept my attention on the individual steps. I didn't have a clear picture of what all those transformations were collectively doing. I needed to zoom out.

Neural networks as function approximators

Function approximation gave me that zoomed-out view. I was already comfortable thinking about models like linear and logistic regression as functions. Given some inputs, these models learn a mapping that produces an output. A neural network can be understood in exactly the same way.

This stripped away much of the apparent mystery surrounding neural networks. Underneath all that machinery, the network was still doing something familiar: learning a function that maps inputs to outputs, just like the other models I already understood. The difference was that neural networks could learn far more flexible and expressive functions. But how do these complex functions emerge from simple neurons?

Neural networks as function sculptors

I already understood abstractly that stacking layers with nonlinear activation functions allowed neural networks to model increasingly complex relationships. But I couldn't visualise it. In Josh Starmer's explanation, each neuron works with a given activation function. The weights and biases slice, flip, and stretch that same activation function into new shapes. As a result, each neuron applies a small nonlinear transformation to its input.

These small transformations and shapes are stitched together across layers to create yet new shapes. A complex decision boundary emerges from many local transformations. Rather than learning one giant function directly, neural networks build complex behaviour by combining many smaller nonlinear transformations. With Softplus, a neuron can contribute a smooth bend into the function surface. With ReLU, neurons contribute piecewise linear changes. Deep networks combine huge numbers of these simple transformations, progressively sculpting a much more complex function.

Don't take my word for it. Watch the video. His animations (and the noises he makes while calculating) make it worth your time.

What clicked for me was that the network didn't need any individual neuron to represent a complicated function. Each neuron could contribute something simple. Complexity emerged from composing and combining many of those simple transformations.

Function approximation had helped me zoom out and see the network as one flexible function. Function sculpting let me zoom in just enough to see how that flexibility could be constructed.

Neural networks as representation learners

One thing that distinguishes neural networks from many of the models I was used to working with is that they can also learn how to represent the input itself. In my previous experience with econometrics and causal inference, deciding how to represent a problem happened largely before fitting the model. I thought carefully about which variables to include, what they measured, and how they should be transformed. The model then learned from the representation I had given it.

With neural networks, deciding how to represent the problem doesn't happen entirely before training. The network also learns useful ways of representing the data. The hidden states in the stack of transformations aren't just intermediate values but intermediate representations shaped during training to become useful for the task.

This is why neural networks are often described as systems for representation learning. This applies even to relatively simple neural networks. For example, a churn model might learn hidden behavioural patterns that help distinguish customers with different risks of leaving. In a larger language model, the learned representations can encode complex semantic relationships. The scale and complexity differ but both are forms of representation learning.

I had been thinking of learning primarily as learning a mapping from my representation of the problem to the target. Representation learning made me realise that the representation itself could be part of what was learned.

Neural networks as optimisation systems

One thing that puzzled me when learning about neural networks was the amount of attention given to activation functions. I understood that without them, stacking linear layers still collapses into a linear transformation. But why did the choice of activation function matter? I initially thought about activation functions through the function-sculpting perspective. Different activation functions give the network different shapes to work with, so I assumed that choosing between sigmoid, tanh, or ReLU was primarily about the kinds of functions the network could construct.

The optimisation perspective gave me a different answer. Unlike simpler models such as linear regression, neural networks generally do not have neat closed-form solutions for their parameters. These parameters are learned iteratively through optimisation. At first glance, optimisation can seem like a secondary implementation detail - merely the mechanism through which weights get updated during training. But it isn't enough for a neural network to be capable of representing a useful function; optimisation has to find the parameters that produce one.

Neural networks were already highly expressive decades ago. The challenge was whether optimisation could actually find a useful solution. Gradient-based optimisation struggled to reliably train deep networks. Much of the progress in deep learning has involved making highly expressive neural networks easier to train.

Many neural network design choices are intended to