A global workspace in language models

Comments

As you read this sentence, circuits in your brain are adjusting your posture, controlling your breathing, and transforming lines and curves on the screen into recognizable words. Most of this processing is invisible to you. But some of what takes place in your brain you do have access to-an image that pops into your head, or a deliberate plan you make about where to go shopping.

Neuroscientists and philosophers sometimes refer to the latter type of brain activity as "consciously accessible," to distinguish it from all the other processing that goes on unconsciously. This activity has special properties: we can describe it, control it, and use it for deliberate reasoning, in contrast to all the automatic processing that goes on without our awareness.

In a new paper, we present evidence that a similar distinction has emerged in modern language models like Claude. We find that Claude has developed a small collection of internal neural patterns that, compared to all its other internal processing, play a special role. We call the collection of these patterns the J-space-named after the technique we used to find them, involving a mathematical concept called the Jacobian.

Each J-space pattern is linked to a particular word. But when one of these patterns lights up, it doesn't mean the model is saying that word-just that the word is on its mind. If you've heard of language models having a "scratchpad" or "chain of thought"-text they write to themselves while reasoning-the J-space is something different. It operates silently, in the model's internal neural activations, allowing the model to think about a concept without writing it down.

Notably, the J-space wasn't designed or programmed by us, but instead emerged on its own during Claude's training process.

We find that the J-space has a number of unique properties, compared to the rest of Claude's processing:

• Claude can report on these representations. If you ask Claude what it's thinking about, it will tell you what's in the J-space. Non-J-space representations are less reportable.
• It can also modulate them on request. If you ask Claude to think about something, or solve a problem silently in its head, it will light up the appropriate patterns in its J-space. By contrast, it has trouble modulating patterns not in the J-space.
• Claude uses its J-space for internal reasoning. If you ask Claude to solve a problem that requires multiple steps, the intermediate steps will light up in its J-space, even when it doesn't say them out loud. These J-space patterns causally mediate its performance in such tasks, despite being smaller in magnitude than other representations.
• Representations in the J-space can be used flexibly for many tasks-for example, once "France" has lit up in Claude's J-space, the model can recall its capital, or its national currency, or the continent it belongs to.
• However, despite its important role, the J-space is not involved in most of what a language model does-speaking fluently, recalling simple facts, using correct grammar, etc. In experiments where we prevented Claude from using its J-space, it still interacted normally, but lost its higher-order cognitive functions.

Our experiments were inspired by a prominent theory in neuroscience that was developed to explain how conscious access works: the global workspace theory. This account pictures the brain as a collection of specialist systems that work in parallel, unconsciously, and largely in isolation from one another. A piece of information becomes consciously accessible when it gains entry to a small shared channel, the "workspace," which is broadcast to other brain systems that can see it and make use of it.

Based on our findings, we think the J-space plays a similar "workspace" role in Claude. For example, we find evidence that Claude's J-space has especially strong connections to the rest of its neural network, allowing it to fulfill this kind of broadcasting role.

None of this tells us whether Claude is conscious in the way people are, or whether it feels anything at all; we'll come back to that question at the end of the post. But whatever its philosophical significance, the J-space is a practically useful tool for us, as it gives us a way to see what Claude is thinking but not saying.

For instance, we're able to use it to catch Claude privately noticing that it's being tested, intentionally producing fabricated data, or pursuing a hidden goal that we planted during training. We've also developed a technique to influence what lights up in Claude's J-space, and thereby influence its decision-making.

More broadly, these findings have changed our understanding of how Claude's mind works, revealing a privileged mental workspace that can be used for deliberate reasoning, operating amidst a sea of more automatic, inflexible processing. Rather than being a chaotic jumble of numbers, Claude's internals have organized themselves in a way that is reminiscent of our own minds.

This post is a short summary of a much more extensive research paper, where you can find more detail on our experiments. We've also released a code repository with an open-source implementation of the core methods, and have partnered with Neuronpedia to provide an interactive demo of our methods on open-weights models. To provide additional perspectives on the broader implications of this work, we also invited commentary from several experts in neuroscience, philosophy, and LLM interpretability, which can be viewed here.

How we found the J-space

The starting point for this research was inspired by one of the key features of consciously accessible thoughts in humans: they can, unlike unconscious processing, often be put into words. If a thought is consciously accessible to you, you can typically describe it if someone asks. We went looking for representations in Claude with the same property: representations that are positioned to influence what Claude might say-not necessarily what it's saying right now, but what it could talk about, if asked.

Our technique is called the Jacobian lens, or J-lens for short. For every word in Claude's vocabulary, the J-lens finds the internal activity pattern that makes Claude more likely to say that word at some point in the future. When we apply the lens to Claude's internal activity, we get a list of words-the contents of the J-space at that moment-which we can simply read.

Claude processes text through a series of multiple internal stages called layers, and by applying this technique over different layers, we can watch these silent words in the J-space evolve as the model works through what to say.

What shows up in the J-space goes well beyond the text Claude is reading or writing. When Claude reads code with a bug that nobody has pointed out, its J-space contains "ERROR." When it reads the raw letters of a protein sequence, the J-space contains the protein's biological function. When it reads search results that are secretly an attempt to manipulate it (an attack known as a "prompt injection"), the J-space contains "injection" and "fake." When we ask Claude a multi-step math problem, the intermediate steps pop up in the J-space, in the right order.

So even though the J-space was discovered by looking for representations that could be spoken, it nevertheless uncovers Claude's internal thoughts. In a sense, this is similar to how some people "think in words," without having to say them out loud.

Claude reports what's in its J-space

Our first set of experiments tested how the J-space is involved in Claude's verbal reports. In one experiment, we ask Claude to silently think of an item from some category-a sport, say-and then name it. If we read the J-lens right before Claude answers, we can see what it picked: "Soccer" is at the top of the list, and sure enough, Claude says "soccer."

By itself, though, this is just a correlation. The J-space might be where Claude's answer comes from, or it might just mirror a decision made somewhere else, like a scoreboard that tracks a game without affecting it. To check, we intervened directly. We reached into Claude's neural network, removed the "Soccer" pattern, and added an equally strong "Rugby" pattern in its place, leaving everything else untouched. Claude then reports that the sport it was thinking of is rugby.

If the J-space were a mere scoreboard-a passive record of a decision made elsewhere-editing it would have done nothing: Claude would still have said "soccer." Instead, Claude's answer followed the edit, which tells us the answer is genuinely read out of the J-space.

In another experiment, we told Claude that a thought might have been injected into its mind and asked it to report what, if anything, it noticed. For instance, in the example below, while Claude was still reading the question, we injected the "lightning" pattern into its J-space. Claude reported that the injected thought was about lightning. The same result held across many injected concepts.

Claude can control its J-space on request

The second property that we tested for was whether Claude can modulate its J-space when asked, like how humans can mentally focus on an image or word. We told Claude to concentrate on citrus fruits while copying out an unrelated sentence about a painting. While it copied the text, the J-space contained "orange" and "fruits," along with words like "thinking" and "imagery" that describe the mental act itself.

We could also ask Claude to do math in its head: when asked to work out 3² − 2 while copying the same sentence, the J-space contains "nine," and then at later layers, "seven." Importantly, nothing about fruit or arithmetic appears in Claude's output, which is just the copied sentence about the painting. The mathematical activity is happening entirely internally, in the J-space.

Claude's control over its J-space isn't perfect. When we told it not to think about something, the concept lit up in its J-space less than when we said it should think about it, but much more than when we never mentioned it. Telling Claude to avoid a thought partly brings the thought to mind, much like what happens to people who are told not to think about a white bear. Claude also seems to notice when its control fails: alongside the forbidden concept breaking through, the words "damn" and "failure" also frequently light up in the J-space, as though Claude is recognizing its own lapse.

Claude thinks in its J-space

In the J-lens readouts above, we saw the intermediate steps of a math problem appear in the J-space. But seeing a concept appearing in the J-space doesn't necessarily mean the J-space is doing the cognitive work. In principle, the real computation might be happening elsewhere, with the J-space just passively reflecting it.

To test whether Claude actually reasons with its J-space, we returned to our swap technique. Consider the prompt "The number of legs on the animal that spins webs is". To answer, Claude has to first figure out that the animal is a spider, and then recall how many legs spiders have. The word "spider" never appears in the prompt or in Claude's answer (it just says "8"); it's a stepping stone Claude uses internally. The J-lens shows "spider" light up partway through Claude's processing, and swapping it changes the outcome: if you replace the "spider" pattern with "ant," Claude answers "6" instead of "8." The second step of Claude's reasoning took its input from the J-space and went along with whatever we put in it.

We saw the same thing in other kinds of thinking. When Claude writes a rhyming couplet, it picks the rhyme word ahead of time, and the planned word sits in the J-space at the start of the line; if you swap it for another word in the J-space, the whole line changes.

We also tested whether J-space representations can be used flexibly-whether one representation can feed many different tasks. This is one of the key properties highlighted by global workspace theory. To test for this flexibility, we gave the model four prompts asking for different facts about France: the capital, the language, the continent, and the currency. Then we swapped "France" for "China" in the J-space, with the exact same intervention in each context. Claude answered with "Beijing," "Chinese," "Asia," and "Yuan," respectively.

In other words, four different downstream computations picked up the same J-space edit and each used it correctly. If Claude stored a separate copy of the country for each kind of question, the edit would have affected at most one of them. The fact that all four answers changed together means they're all reading from the same shared representation, which is what a workspace is for: information gets written in once, and many different systems can use it.

How can one representation of a concept serve so many different tasks? Earlier, we mentioned that the J-space appears to be wired up to the rest of Claude's neural network especially densely. For any activity pattern, we can measure how strongly the various components of the network are connected to it-how many of them are positioned to read information from that pattern, or to write information into it. J-space patterns stand out dramatically on this measure: far more components read from them and write to them than for ordinary patterns, in some parts of the network by a factor of about a hundred. This is the kind of wiring you'd expect of a broadcasting hub, where many systems post information and many others pick it up.

Claude's automatic processing skips the J-space

In humans, most of the brain's processing is not conscious-we don't deliberately think about parsing grammar while reading, or balancing our bodies while walking. Similarly, we found that most of Claude's processing doesn't involve its J-space. It turns out that the J-space holds only a few dozen concepts at a time, and accounts for less than a tenth of the overall activity in Claude's internal processing. So what is all the rest of the neural activity doing?