Neuron-Level Explanations and Their Limits

The obvious way to understand a neuron is to find the inputs that make it fire hardest and read them. It is also the way that produces confident, wrong explanations, and the fixes - automated explanation with automated scoring - are interesting precisely because of how they are scored. Where this starts: max-activating examples Run a large corpus through the model, record one neuron’s activation at every token, sort, and look at the top few hundred. Sometimes the answer is immediate: a neuron that fires on quotation marks, or on tokens inside URLs, or on the word after “Dr”. The multimodal neuron work at OpenAI (Goh and colleagues, 2021) is the most striking result in this genre - units in a vision-and-language model that responded to a concept across photographs, drawings and the written word for it. That is a real finding about representation and it is why the approach stayed popular. Why top examples mislead Three distinct problems, and they compound. You are reading the tail. The top 0.01% of activations is not a description of a neuron’s behaviour. A unit can look like a “French text” neuron in its top hundred examples and fire moderately, across a much larger set of tokens, on something else entirely. Since contribution to the output is roughly activation times its outgoing weight, moderate activations on millions of tokens can matter more than extreme activations on a few. Human pattern-matching is unreliable and unfalsifiable. Shown twenty examples with any shared property, a person will find one. Without a held-out prediction, the explanation cannot be wrong. The interpretability illusion. Bolukbasi and colleagues demonstrated in 2021 that the apparent meaning of a neuron can depend on which corpus you look at - the same unit yields different clean-looking explanations from different datasets. Neither explanation is a lie about its own data; both are incomplete, and nothing in the procedure tells you so. The automated pipeline OpenAI published the standard structure in 2023 (Bills and colleagues): use a language model to explain another language model’s neurons, and - this is the load-bearing part - use a language model to score the explanation. - Collect. For the target neuron, gather text excerpts with per-token activations, including high-activating examples and a random sample across the range. - Explain. Show an explainer model the excerpts with their activation values and ask for a short natural-language description of what makes this neuron fire. - Simulate. Give a second model the explanation and a fresh, held-out excerpt without activations, and ask it to predict the activation at each token. - Score. Correlate the simulated activations with the real ones on the held-out text. That correlation is the explanation’s score. The elegance is in step four. An explanation is now a predictive hypothesis tested on data it was not derived from, which is exactly what a human reading twenty examples fails to provide. It also gives an automatic ceiling test: run the pipeline on random units, or with deliberately wrong explanations, and see what score noise achieves. How an explanation is scored Worth dwelling on, because the scoring choices determine what the numbers mean and they are easy to get wrong. - The held-out sample must span the activation range. Score only on top-activating excerpts and every explanation looks good, because on those tokens almost anything predicts “high”. The discriminating cases are the tokens where the neuron does not fire but a naive reading of the explanation says it should. - Correlation rewards getting the shape right, not the scale. A simulator that predicts the right pattern with uniformly wrong magnitudes scores well. Whether that is acceptable depends on what you want the explanation for. - The simulator is a confound. A better simulator raises every score without any explanation improving. Scores are comparable within one pipeline configuration and not across different ones, which makes cross-paper comparison of these numbers unsafe. - A low score is ambiguous. It can mean the explanation is wrong, or that the neuron is genuinely polysemantic and no short description exists. Those are very different findings and the score does not distinguish them. The other half: what the neuron writes Everything above describes a neuron by its inputs - what makes it fire. That is only half of what a unit is. The other half is its output: the column of the MLP’s down-projection belonging to that unit, which is the vector it adds to the residual stream whenever it activates. That vector can be read directly. Push it through the final layer norm and the unembedding, exactly as the logit lens does, and you get the tokens this neuron promotes and suppresses when it fires. No corpus, no sampling, no explainer model - it is a property of the weights. import torch # GPT-2: c_proj.weight has shape (d_mlp, d_model), one ROW per MLP unit W_out = model.transformer.h[LAYER].mlp.c_proj.weight # (d_mlp, d_model) direction = W_out[NEURON] # what this unit writes with torch.no_grad(): logits = model.lm_head(model.transformer.ln_f(direction)) top = logits.topk(10) bot = (-logits).topk(10) print("promotes:", [tok.decode([t]) for t in top.indices]) print("suppresses:", [tok.decode([t]) for t in bot.indices]) The two halves should agree. A unit whose max-activating examples are all legal text and whose output direction promotes legal vocabulary is a coherent story from both ends. A unit with a clean input story and an output direction that promotes nothing recognisable is telling you the description is incomplete - most likely it feeds a later computation rather than the output, in which case the honest description is about what it contributes to, not about what it means. Check the orientation of the weight matrix before trusting any of this. GPT-2 uses a Conv1D layer whose weight is stored transposed relative to a standard nn.Linear , so the axis indexing a unit differs between architectures. Verify by confirming that the vector you extracted has length d_model , not d_mlp . Getting this backwards produces a plausible-looking list of tokens for a vector that means nothing. The three structural limits Polysemanticity. If superposition is right, many neurons participate in several features and have no single meaning. An explanation pipeline will still produce a sentence for such a neuron, because it always produces a sentence. The low score is the only signal, and it is a weak one. Explanation is not causation. A perfectly predictive description of when a neuron fires says nothing about what the model does with it. A neuron could fire reliably on a well-defined pattern and have almost no downstream effect. The causal question needs an ablation, and the two questions are routinely conflated. Coverage. A large model has hundreds of millions of MLP units across its layers. Explaining a sample tells you about the distribution of explainability; it does not give you a map. And the neurons you would most like explained - the ones implicated in some failure - are not the ones a random sample surfaces. Features instead of neurons The field’s response has been to change the unit of analysis. If the neuron basis is the wrong basis, find a better one: train a sparse autoencoder on the layer’s activations and interpret its units instead. Reports from this line of work describe substantially more monosemantic units than raw neurons give. The explanation pipeline transfers unchanged - collect, explain, simulate, score - and the scoring is more important here, not less, because a sparse autoencoder produces its decomposition whether or not the model has one. A high explanation score on autoencoder features plus a causal check that intervening on a feature changes behaviour as its description predicts is the combination that means something. Either alone does not. What to do if you must interpret a neuron - Sample across the whole activation range, not just the top. Include the median and the zero region. - Write the explanation as a prediction, then test it on held-out text: generate inputs your explanation says should fire and inputs it says should not, and check both. - Check the outgoing weights. What does this unit write into the residual stream, and which tokens does that promote when passed through the unembedding? A neuron whose description and whose output direction disagree is telling you the description is incomplete. - Ablate it - set it to its mean over a distribution rather than to zero - and measure what changes. If nothing does, your explanation may be correct and irrelevant. - Report the score of a deliberately wrong explanation on the same neuron. That is your noise floor, and without it a number like 0.4 means nothing. Top comments (0)