Neural Networks Are Already Clean

The Short Version

I think I have good news about the nature of neural networks.

After many months studying neurons and circuits in image and language models, I’ve come to believe a network’s own mechanisms are its legible building blocks, and that the outlook for mechanistic interpretability may be far better than I’d feared.

There are at least two ways to approach mechanistic interpretability. The first is the simplest: look at a network’s neurons and circuits, study how they relate, watch data flow through them, and run experiments to test explanations. The second starts from a list of hypothesized features and tries to extract them all, checking that each is clean and none are missing. The first is just rapid normal science on the original: building understanding from many specific experiments on the network itself. The second tries to design increasingly sophisticated reconstruction objectives to capture these hypothesized features, and usually discards or ignores the original in favor of studying the reconstructed list.

The field began with the first approach and fell back to the second because neural networks seemed to be tangled knots. We had canonical examples of polysemantic neurons, the broad name for a neuron that appears to have multiple unrelated jobs or that you can’t understand at all. The most famous one, in an image model, responded to both cat faces and the fronts of cars. And the last layer of the early field’s model organism, InceptionV1, seemed so full of arbitrary mixtures it wasn’t clear how to study it: almost none of the neurons were easily interpretable.

The leading explanation for polysemanticity became that neural networks use two learning principles. When a network has enough neurons for what it has to learn, it groups things that share a pattern useful to its downstream readers. This is generalization, and in early image model organisms, roughly the first half learned this way. But when space runs out, the network is thought to switch to a second principle, superposition, folding unrelated things together simply because they don’t interfere. The result is a kind of knot: the network could have folded in many almost equally good ways, so which things a neuron groups together tells you almost nothing. You have to unfold the knot before you can study anything. In practice that means training a second model, a decomposition such as a sparse autoencoder, to approximate the network as it would look unfolded and studying that instead. If the network really is knotted, there may be no alternative.

The first main result is that superposition was the wrong explanation for InceptionV1. Architectural problems made later layers overflow. Weak interpretability tools amplified these problems, making the late layers seem incoherent. Architectural designers with little interest in interpretability have since improved things, with now-ordinary model pieces like a residual stream, originally aimed at helping deep layers learn better. With these ordinary improvements, and no change to model size or training data, 99–100% of neurons in that corresponding last layer are easily interpretable, where at the time almost none seemed to be, and the model is broadly legible the whole way through. It’s the first non-trivial neural network I feel I mostly understand; you can browse it here. I trained versions of it across a 300x range in size, and it’s broadly legible at every scale. With less room, the network doesn’t fold unrelated things together; its neurons just get broader, like one car neuron covering many kinds of car. With more room, that neuron differentiates into many specific cars. So the problem wasn’t the presence of a second learning principle, it was just an ordinary learning failure of the first one, and it’s easy to fix.

The canonical vision examples of polysemanticity and superposition were errors too. In 2019, I found the circuit that was originally introduced as the superposition phenomenon in Zoom In, and others helped me interpret it. It showed a car feature appearing to spread into a set of unrelated dog neurons. It was the only one we showed, and the only one I ever saw. We read it as the network placing the car in a high-dimensional hiding place, for downstream neurons to read back out. But nothing reads that combination. It wasn’t hiding anything; it was ordinary generalization and composition. To my knowledge, no one has published a second circuit of that shape. The main candidate I could find is a “social security” direction found with sparse probes in Finding Neurons in a Haystack (2023), but the model doesn’t use that direction, and the mechanism it does use, which I mapped out, is again ordinary generalization and composition.

Modern language models are also far more interpretable than generally believed. Anthropic’s Towards Monosemanticity (2023), studying a small one-layer model, gave the median neuron a 0 on its rubric out of a possible 14, meaning the rater couldn’t form any hypothesis. The automated interpretability project I started at OpenAI (also 2023) found about 1.7% of neurons interpretable by our most generous measure.

In our current audits of Qwen3-32B, like the 1000 neuron audits we have in this paper, we estimate that roughly ~90% of neuron poles have one main job, often with small related offshoots, another ~7% have two related jobs, and only ~2-4% are arbitrary mixtures.1A note on this number: the share of neurons with one job varies depending on how you set the threshold for counting weak activations, depth, density of sub-circuits, and what you view as “one thing”.

How far to go into weak activations is a big factor here. There’s a section describing weak activations below. It tends to increase how many “related things” there are, more than whether the neuron is a judgement of something legible. For instance, weak Trump neurons brings in MAGA hats and people related to him. Weak Titanic neurons bring in stories of more general voyages, or steam engines and other technologies of the era.

There’s also the question of whether to care about activation percentiles, or fraction of the strongest activations. In general I think what the neuron learns should be trusted, its natural sparsity and activation ranges and so on. Which I used to mostly not believe! I’m somewhat against trying to standardize things across all neurons. Percentiles are more common today, I think because they’re easier to think about.

It goes the other way too. For instance, Titanic neuron when you look at its “unrelated jobs” it’s often something that detects “boat” next. But the tool when it judged context looked backwards. When we look at these “unrelated jobs” in depth, it’s almost always this kind of thing. It’s easy to build an imperfect tool.

Errors in these scores go the other way too. But overall, I don’t think coming to a single score is actually that important. But when it was taken out, early reviewers felt unanchored without even a rough number. In the sets I’ve looked at more closely, considering strong activations, it’s closer to 97% non arbitrary mixtures, but these are relatively small samples.

Overall distilling everything into one “cleanliness” score should be taken as a broad pointer to the regime a neural network is in, rather than something precise to either optimize directly or hold too tightly. A lot of this essay is for being less dimensionality reduced, treating each neuron and circuit differently, according to their motifs, depth, role etc. Despite this, a rough number can be useful.

What I think matters much more is the generalization or superposition question. Since these cases appear to be recursive generalization, where the branch forms “a second thing” is far less important. But these scores for this model are with settings I thought were most productive for studies of motifs and different types of audits I’ve done. More details will be in the technical companion paper. Dense and Mixture of Experts models seem roughly similar. These compress a high-dimensional picture into a few numbers, which shift with depth, scale, and how closely related two things must be to count as one, and the errors run both ways. A lot of what I’ll advocate for in this essay is not doing compression, treating each motif separately, and I don’t think model-wide general cleanliness scores like these should be held very tightly, but it’s helpful for having a rough anchor. Lighter audits find similar results in models up to trillions of parameters, like Kimi K3, and the companion paper will have more detail. A lot of the “one thing” question comes down to questions like whether you consider two closely related idioms across two languages one thing or two things, which I don’t think much in interpretability hangs on. What matters a lot more is why things share a neuron in the first place.

Most of the remaining apparent polysemanticity seems to be a motif I call recursive generalization. It’s simple: as a neuron grows, it sprouts offshoots the way a tree branch sprouts sub-branches. For instance, one neuron that detected “iy” grew to detect “diy,” and then “do it yourself,” which doesn’t contain “iy” at all. It first appeared in a forgotten subsection of a paper I worked on that was mostly about something else, Multimodal Neurons (2021). At the time I hadn’t watched neurons grow. Now it’s easier, and it holds up everywhere I’ve seen. Even the original InceptionV1 wasn’t knotted. When I retrain its original architecture, which still produces messy deep layers, and follow the worst neurons I can find, they turn out to be overgrowing locally into reasonable, adjacent territory.

So these were two explanations for polysemanticity. I helped propose both, and in some ways they’re opposites, since generalization groups similar things together and superposition groups unrelated things together. Superposition is the one the field bet on, and until recently, so did I. But I now think recursive generalization was basically right, which leaves one learning principle after all. It’s still generalization, but it turns out generalization is fractal. That’s far less concerning: there’s something to learn from the way things were grouped, and nothing to learn from two items in superposition. And since the generalization itself is fractal, our explanations can be too: start with a basic explanation, then zoom in when you want to understand more. In that sense, almost the entire network is interpretable if we look closely enough, especially if we look at how it grew over training.

Why was the gap so large? I think several ordinary problems compounded. The explainer models of 2023 couldn’t run the loop of conjecture and test; the tools and metrics were weak, rewarding explanations that could simulate activations rather than ones that were right, which often come apart; no one knew what language neuron motifs to expect, and people broadly looked for the wrong kinds of things; and most studies used small or simplified models. In a sense, those models really had run out of room, which made superposition the natural explanation. But a network short on room doesn’t switch learning principles; its neurons just get broader, as in the small vision models. Broad neurons are usually easy to break down once you know what to look for. Without that knowledge, and with primitive tools, they look like arbitrary mixtures.

Amplifying all of this is that interpretability has a surprising amount of momentum, the way a jigsaw puzzle or crossword does as you fill it in, since everything starts constraining everything else. At first, you’re looking at a single neuron, and if you can’t understand it, little else helps: circuits and attribution graphs only explain a neuron in terms of other neurons you don’t understand yet. This can make getting started hard. Once the map starts to fill in, though, it goes surprisingly well. If you don’t understand exactly what a neuron does, you can study what it connects to, the ecosystem around it, and, in a mixture-of-experts model, what its expert specializes in. As with a puzzle coming together, every piece comes with many clues, and they’re well structured and easy to run experiments against. That’s why nearly empty and mostly filled in aren’t surprising places to be; they’re the natural anchors.

Take a neuron that tracks a sense of righteousness when a bargain purchase turns out to work better than the expensive alternative. Neurons that track a feeling or social cue like this are some of the most common we’ve found. Imagine finding this one with no knowledge of language neuron motifs and no circuits to lean on, because few neighboring neurons are explained yet. Most humans and agents expect neurons to detect nouns by firing on them, which they mostly don’t. Look at where this one fires most strongly, and you see a pile of seemingly unrelated commas.

Now imagine the map is mostly filled in. You know many neurons detect subtle feelings or social contexts, and that they often fire not on the words that carry the feeling but on punctuation, like commas, where the model summarizes what a sentence meant, or on the word just before the feeling matters for predicting what comes next. So you read the surrounding sentences instead, and check its steering effects (what the model writes when you turn the neuron up), which line up closely. You look upstream at how it’s built and downstream at what it does, and both line up. You check its negative pole, what pushes it strongly negative, and find a subtle variant of the same cue, or the moment the social context ends. You can run specific tests, like checking which store brands it responds to. Any of these could break the explanation, yet they keep pointing to the same place. And understanding this neuron sharpens the broader motif: after thousands of subtle-feeling and social-context detectors, you start to know what to expect from the next one, which is part of why the map fills in faster as it goes.

It works on the tools too. When you can see 2% of a network, you can’t tell where your tools are broken; when you can see most of it, the unexplained remainder points straight at them. That’s how some coworkers and I found a double-negative regime in SwiGLU neurons that, as far as we know, no one had noticed, and that a decomposition mixing many neurons together would likely have hidden.

Today, reaching that initial momentum is much less of a problem: the models are easier to study, the automated scientists are far better scientists, and we know more about which motifs to expect. Explanations don’t have to be perfect, either. Crossword solvers often pencil in two candidate words early on; as the crossing answers fill in, one usually wins, and sometimes a crossing rules out both. An uncertain explanation works the same way: as everything around it gets solved, it becomes increasingly constrained. The bad news about modern neural networks is that they’re enormous. The good news is that the enormity helps: as the map fills in, every explanation is tightly constrained by everything around it, and bigger models learn narrower neurons that are easier to recognize in the first place.

In this sense, understanding the whole network may be easier than understanding just the parts we care about most. First, it’s hard to know how to think like a neural network until you’ve studied it closely, so we don’t know precisely which questions to ask. For instance, "deception" is a useful initial frame, but the building blocks that contribute to deceptive behavior by models probably have alignment-relevant microstructure that will be hard to know without going in and looking. Second, targeting a small set of mechanisms is like working on one row of an otherwise empty crossword. It’s doable, and might work, but without the crossings it’s hard to know whether you’re right, so it may be easier to solve the whole thing, or at least the broader parts of the network these mechanisms connect to.

When I ran this process on modern convolutional networks, it basically worked. They can map out neurons and circuits, build explorable microscopes and explore them, test explanations thoroughly, track hundreds of thousands of attribution graphs. There was never a point where normal science broke down. I now feel I basically understand these models, certainly at least enough to know where I’d start asking any question I had; eventually, the crossword just got filled in. So far it has worked in language models too, up to trillions of parameters. Those aren’t close to solved, but I’m optimistic about scaling the process.

All of this has brought me back to an old hope: understanding a neural network directly, by studying its mechanisms and the data flowing between them, without first trying to fix or untangle it. Put together, the argument has four steps. Networks use one learning principle, not two (if not everywhere, then nearly enough that it doesn’t matter for study), so there’s no blocker to beginning ordinary science on them. With modern models and tools, that science goes much better than most people expect. And understanding compounds, so it gets easier as the map fills in. At minimum, then, a network’s own mechanisms should be part of the toolkit. The fourth step is highly speculative: this might be most of what we need, and interpretability might be far more tractable, possibly in the near term, than we’ve assumed.

That’s been the pattern for deep learning in general: the simple thing at absurd scale beats the clever one. Rich Sutton called itthe bitter lesson. Maybe there’s a bitter lesson for interpretability too: scaling ordinary investigation beats engineering a new decomposition.

A technical companion paper will offer a narrower and more detailed account of language model neuron experiments, such as what we’ve found about the different regimes of modern non-linearities and what kinds of circuit motifs show up at different depths. This essay glances at many places and conjectures about where it all might be pointing.

Cars and Animals

The most famous polysemantic neuron is 4e:55 of InceptionV1, which looked like an arbitrary mixture of cats and cars. It’s been brought up as evidence that neurons might not be the right unit of study in several of the field’s most widely read papers, from Feature Visualization in 2017 to Zoom In.. In Zoom In, which I co-authored, we said outright that it “isn’t responding to some commonality of cars and cat faces”.2In Multimodal Neurons we came very close. We asked whether the neuron might be responding to some shared feature, “perhaps cat eyes and car lights look similar,” then decided from the feature visualization’s whiskers and chrome that it was “genuinely OR(cat, car).” We were right to look for a shared shape, but I think we underestimated how far generalization reaches. The shared feature was the V. I think the whiskers misled us because generalization happens inside high-level patterns. The V neuron really does take in cat circuitry, and feature visualization’s job is to raise the activation, so once it finds the cat route it adds everything that route brings. The whiskers come along for the ride, even if the neuron doesn’t “want” them. Downstream, the car readers do get a little cat signal through the V neuron, but it barely reaches the output: one cat detector firing does little on its own, since many work together, so the network never needs a negative weight to cancel it. To see the shared pattern, you have to visualize what the neuron’s inputs have in common, not the neuron on its own.1

After months of seeing clean neurons I wondered if we’d misinterpreted some early prominent examples of polysemanticity and superposition wrong. The canonical polysemantic neuron seemed like the obvious place to check, so I opened up the cat and car image grid I’d seen hundreds of times before and did something I’d never done: I spent a few seconds wondering what one thing this might be. Almost instantly it was obvious: it’s not a cat and car neuron, it’s a V-shape neuron.

I looked at its circuit connections: V-shaped flames, V-shaped water jets from a fireboat, V-shaped fox faces. I looked at its negative pole: an O, round where the V is pointed. Why would an arbitrary mixture of cats and cars need to connect so strongly to a flame?

inputs

4d:493 fox snout

4d:477 wolf face

4d:201 van front

4e:55

Pointed→generic

Round→generic

readers

5a:94 flames

5a:352 coyote face

5a:317 martini glass

Pointed makes it fire

1. dog
2. fox
3. cat
4. car
5. pool table
6. bullet train

Round pushes it negative

1. dog
2. dog
3. snail
4. golf ball
5. tennis ball
6. clock

Then I went to the canonical superposition circuit I first found in 2019. The V neuron and this circuit were Zoom In’s two main examples of why its picture of neurons, that nearly every one turns out understandable if you look seriously, might be “overly rosy”.

The circuit appeared to show a car going into a bunch of unrelated dogs, as if the dogs were a kind of high-dimensional storage place where the car could be recovered later as needed.

But that’s not what the car was doing. Cars and dogs were being brought together because of shared pose. InceptionV1 has more than a hundred of these generalized pose neurons, each one its own clean detector of a pose shared across cars, dogs and other objects, and to my knowledge no one had noticed them. Later layers keep building on them, all the way to the end of the model. The car was not being stored in an obscure place, and nothing directly reads out the combination. It’s the opposite: it’s continuing to differentiate into car specialists and finer poses.

Part of the reason was not knowing about generalized pose neurons, and part of the reason is that the technique we use to render icons of images — feature visualization — ends up being reward hacked in deeper layers. The intent is to find an image that maximizes the neuron, but the way that’s done is through gradient descent, and it takes the easiest path, which in InceptionV1 often means dogs. There are lots of dogs in InceptionV1’s training data and a lot of dog neurons, and the early optimization falls into them and never recovers. If you even lightly bias it towards the natural paths the top images take, most of them are cars, and some are both dogs and cars, because of shared pose.

What we thought

Arbitrary storage place

What’s going on

Just cars

Shared car and dog pose

It can be tempting to discount this generalization, under the assumption that the dogs and cars are only together because there’s not enough space. Maybe it “really wanted” separate dog-pose and car-pose neurons, and a bigger model with more space would have separated them. It’s true that bigger models tend to have more narrow neurons, but when networks scale up, they can do it in counterintuitive ways.

Looking at DINOv2, which is more than ten times the size and about ten years of architecture later, it has more pose neurons, but many of them are still generalized. I’ve seen this in every vision model I’ve looked at, of any size.

DINOv212× InceptionV1

ConvNeXt large4× InceptionV1

ConvNeXt medium1× InceptionV1

ConvNeXt small1/75× InceptionV1

Both human and automated interpreters seem strongly biased toward describing neurons as detectors for groups of nouns, like cats and cars, rather than for a shared pattern, like a V. Sometimes a neuron really does respond to a group of nouns. In my experience, though, it’s more often a shared pattern, and the neuron is better understood as a verb than a noun. Its job isn’t to name what’s in the input but to draw a distinction downstream neurons can use: it separates two subtle patterns at its positive and negative poles (a V versus an O) and pushes the flow of activations toward different neurons accordingly.

The bias is strong enough in me that I can often unblock my understanding of a neuron by asking: if I couldn’t use groups of nouns at all, how would I describe this neuron to someone else? Usually the shared pattern pops out right away, and it’s easy to test on more groups of nouns (for the V: shinkansens, pool tables). I suspect this is the largest source of misattributed polysemanticity in mid-layer neurons. Many explanations I’ve seen that call a neuron polysemantic look to me like a single pattern described as a list of nouns.

This is what I think we should do with a neuron that means two or three things: treat it as one neuron with a few grouped meanings, rather than splitting it. The grouping probably wasn’t a problem for the network: the meanings sit together because, from the network’s point of view, they’re similar. That makes the grouping a clue to how the network thinks, and following it can turn up generalizations we didn’t know about, as cats and cars did here. Often the extra meanings also melt away on closer inspection, turning out to be minor and rarely active. Finally, some apparent meanings are just our tools being bad, and this essay has several examples: the double-negative regime of SwiGLU, an unread saliency knob that makes the residual stream hard to visualize, and the reward hacking in feature visualization we just saw. Keeping the original neuron whole keeps these errors visible, so the error correction can continue. If we’d used optimization to get a cleaner list, we might have lost many of these hints.

[footnote on network: this is part of why I think superposition vs recursive generalization matters a lot. In both versions, it wasn’t a problem for the network. But in one version it’s because two things were unrelated, and in the other because they’re related. I think we’re in a world where it wasn't a problem for a network, and there’s something to learn from that, because they’re related.]

A History of Microscope Errors

The story of interpretability, as I see it, has largely been one of finding out that what was under the microscope was better than we thought. The microscope just wasn’t tuned right, and what needed tuning was often pretty subtle.

Modern neural network interpretability arguably started with feature visualization, which renders a synthetic image of what a neuron responds to most: the famous psychedelic pictures. For a while they didn’t look very good. You could reasonably conclude the neurons had learned something, just nothing very clean. It turned out the method was missing transformation robustness. Once researchers added it, the same neurons came out crisp.

But that only gets you so far into the model. Near the end, feature visualization starts to break down again, for two reasons: the reward hacking we saw with the car circuit, and regularizations that weren’t quite right. Fixing it didn’t require anything fancy. A few subtle tweaks to reasonable settings, and now we can see much deeper into the network. Same neurons, better microscope.

2009-style20172026Strongest photos Middle layer

Strongest photos

Late layer

Strongest photos

But it wasn’t all the microscope. The model needed to be tuned too. InceptionV1, the early model organism of the field, was totally uninterpretable in the late layers. Liv Gorton said “To the best of my knowledge, there are essentially no interpretable neurons on InceptionV1’s final layer,” which is close to my own recollection, I’d have maybe said 5%. This is what they looked like.

The leading explanation as far as I knew was that there were too many high-layer features needed to be represented for the number of neurons and parameters there were, so it started packing ones that don’t belong together arbitrarily, since they don’t collide. The early layers were monosemantic because not as many early layer features needed to be represented.

But this wasn’t the right explanation. Here’s a more modern convolutional network with the same number of parameters and dataset, but with residual connections, LayerNorm, and a couple other improvements. The last layer is beautiful.

7.06M · parameter-matched28.6M

‹

LadybugsTerriersPlatypusMilk cansYellow on greenAltarsAcorn squashCrocodilesGrand façadesMarmosetsTape playersRabbitsWarthogsLacewingsStinkhornsHalf-tracksPoodlesCatamaransCurled horns

›

Ladybugs · Neuron 127

Feature visualization
Open neuron ↗

leaf beetleleaf beetleleaf beetle

So two things were required. First, the neural network itself needed the ability to learn well, using an architecture built by people with no interest in interpretability because it helped the model learn better. For instance, a residual stream, which was introduced to help neural networks learn deeper, and also seems to help the deeper layers be clean, which may be related. And secondly we needed tools strong enough to see them clearly.

The last layer of InceptionV1 was one of the main pieces of evidence that neural networks were almost fundamentally uninterpretable, since the way they learn overlaps unrelated mixtures. Replacing the neurons seemed like the only option. I put it around 5% with tools of that era, others at ~0%. With better interpretability tools (fixed feature visualization, methods for visualizing attribution etc) it’s more like 50% on the same network. This is all with the 2014 model whose checkpoints and training details were lost to time. Retraining the same architecture with more reasonable modern regularization it’s more like 70%. In the ConvNeXt of the same size, it’s about 99%. They’re explorable here.

Recursive Generalization

So we’ve shown the last layer was fixable. But what was it doing before it was fixed? Even before the architectural improvements, it wasn’t putting things together arbitrarily. When we retrained the model and watched a neuron form, it started on yellow flowers, in photos that sometimes included a dog. Then it grew along each facet of those photos: toward other kinds of nature, toward green objects like balls, and toward dogs without any flowers. This is true of every neuron I looked at. The improved architecture shows the same pattern, but much less of it.

We call this recursive generalization: a neuron generalizing along more than one axis, often first along one axis, then sprouting outwards in ways that are easy to follow locally, but stop it from being “monosemantic” in the end.

For instance, a language-model neuron responds to “iy,” including in “Miyazaki,” then to “diy,” and then to “do it yourself,” which no longer contains “iy” at all.

There are examples all over the place in both image and language models. Like a geology neuron that generalizes into Cesium, a geology javascript library.

Usually it’s very obvious what happened, to an extent where it’s hard to know whether to just call it normal generalization or not. Like an enemy neuron that fires for a Russian poem about enemies. Sometimes it’s less clear but still plausible. A Srinivasen neuron also lightly fires for the UniProt protein database. The project has several Srinivasen’s associated with it, but it’s hard to know for sure.

We’ll talk about these recursively generalized offshoots and how much they matter in the language model section. Here we just want to talk about what the motif is.

We first hinted at this in Multimodal Neurons, calling them “conjoined features.” I’d forgotten until I went back through the field’s most famous models, months into studying recursive generalization. As far as I can remember, no one, including me, suggested it might be the general explanation, for instance for what was wrong with InceptionV1’s last layer. Without training checkpoints we couldn’t watch it form, but the explanation was basically right. The appendix explains why I think the new name is more accurate.

Superposition

Recursive generalization isn’t a particularly bold explanation, it’s arguably just what we should expect out of ordinary generalization. But there’s also a much bolder explanation that has been proposed for polysemanticity, a new learning principle.

The main learning principle we see everywhere is generalization. With generalization, things are put together because of a shared pattern that it’s useful to have in one place, so it can be built on. With superposition, unrelated things are placed together because they don’t interfere. These have opposite predictions: generalization should be local, superposition should be non-local, since it would want very different things to be near each other.

Generalization would want a floppy ear dog neuron, which can be thought of as lots of slightly different floppy ears at different angles and kinds together. Superposition would want a floppy ear dog + Latin poetry neuron.

The definitions get tangled here. Along with superposition, Zoom In introduced the idea of features as the fundamental unit of neural networks. Depending on where you look, superposition then means either “more features than dimensions,” which even a fully monosemantic network would satisfy, because its neurons compose, or a learning principle like the one above, applied sometimes to what neurons learn and sometimes to where features go.

So “superposition” is itself in superposition between several common uses. We focus on the learning principle applied to the mechanisms like neurons, because whether it forms holds such a disproportionate share of the good or bad news for interpretability. It’s also fairly easy to test: recursive generalization and superposition predict nearly opposite things, and you can watch a neuron grow and see which one happens.

When I say superposition in this essay, I’m referring to this version, and we’ll address the question of “more features than dimensions”, which I basically don’t think is bad news, later.

Superposition of the Gaps

Polysemanticity is often more a claim about the observer than the neuron. It says there’s no good explanation yet. There’s a few reasons that could be:

1. Bad tools: our lens or microscope isn’t good enough
2. Bad explanations: created from those tools
3. Bad learning: the network learned poorly due to bad architecture, training data, or something else
4. Recursive generalization: while generalizing, the neuron branched off in a few related directions
5. Superposition: a change in learning principle where the network packs unrelated things into the same neurons because they rarely collide.

For years I was a staunch superposition believer. I started studying language model neurons not out of any interest in neurons or superposition, but because I was interested in whether agents had research taste! I wanted to see if they’d ask good questions as they tested explanations, and started with neurons only as a baseline. I expected them to run into intractable feature fragments in superposition. That would produce a new baseline for how clean neurons really are, then the agents could continue with some cleaner learned basis. But it never came, because the neurons kept coming, resolving to a simple explanation that made surprising predictions that held.

In a frontier language model, I found a neuron that mostly fired on “pr”, but also on a bunch of Base64 URL strings as well as some Bengali. I thought this must finally be an arbitrary mixture. But decode the URLs: “pr” words. Translate the Bengali: “pr”.

In a dense Qwen model, a coworker came across a neuron that standard top-k automated interpretability lists as a “polysemantic comma neuron”. But looking more closely, it turned out to track Colorado: phone numbers, slang, and connections to other Colorado neurons. Steering it made the model want to go hiking. The commas were where it stored a running summary of the Colorado context so far. In general attention seems to collect on the punctuation, which we call the summary motif. Once you know it, a lot of similar neurons get easier to understand.

Other times, we just didn’t have enough data. At first we didn’t include long context, two paragraphs felt like enough. But this accidentally made all the long-context neurons look polysemantic.

In another case a neuron looked like an arbitrary mess, with nothing firing strongly. It turns out it looked for specific signs within pages of health SEO spam, and once we had a sufficiently broad dataset, it was understandable.

Across hundreds of cases like these, nothing shook my belief in superposition. It just kept moving. If the strong activations came back clean, superposition would move to the weak ones, or to combinations of neurons, or to the neurons I hadn’t looked at yet. Or maybe across layers themselves.

But eventually I wondered what was going on. Unless this was some dramatic extended case of selection bias, something wasn’t matching the early model organisms. So I went back, the original evidence fell, and that led to this essay.

I now think I was making superposition the null hypothesis when it should have been the last hypothesis, since it’s the most radical and the least testable. It just looks like things that don’t make sense. But that’s also what bad tools look like (feature visualization reward hacking), or recursive generalization you haven’t followed yet (“iy” neuron), or a bad architecture (5b of InceptionV1), or generalization you haven’t understood yet (cat and car).

I think part of what was going on was that superposition just felt right. Anthropic’s Softmax Linear Units paper said as much in 2022: “In the mechanistic interpretability community, superposition is often treated as the default hypothesis simply because it seems intuitively more compelling than other explanations, but there is little evidence.”3In the next sentence, the paper says its own results are moderate evidence for superposition. That evidence is circumstantial, in the paper’s own word: SoLU suppresses activations spread across many neurons, an extra LayerNorm restores performance, and after the LayerNorm a neuron’s weak activations looked less related to its main job, which the paper reads as features being smuggled through small activations. We read it as evidence about distributed representation in an architecture built to suppress it, rather than about neurons packing unrelated things together, which is the sense we care about. Telling the two apart in weak activations is hard even with today’s tools, since meaning shifts in every direction at once as activations weaken, and it was harder still in 2022, for all of us. The paper also notes that its discussion of superposition has been superseded by Toy Models of Superposition, which we address in the appendix. We bring it up not as a criticism of either paper, but because it’s the only place we know of where someone said plainly how little real-world evidence there was, and we think people still often assume there’s more than there is. To my knowledge, no one has found a superposition circuit in a real network since my original one, which was an error. I don’t see anything like it in real models, despite expecting to, and I think the bar for claiming it should be high, since it’s such a radical explanation and would be big news, terrible news, for how we should study neural networks.

Of course, this risks the opposite error, an explanation of the gaps: assuming every neuron must have a sensible story. Two things make it less worrying. First, each of the other causes can be tested. A better tool either resolves the neuron or it doesn’t; retraining with a better architecture shows whether learning was the problem; and recursive generalization can be watched as it grows, in small steps. Superposition’s only symptom is that something doesn’t make sense. Second, real explanations tend to be narrow and easily testable. Once you suspect a neuron is a Colorado neuron, steer it and see what it says. If a neuron fires for some narrow TV trope, the set of things that could count as that explanation is tiny, while “something that doesn’t make sense” covers everything outside it.

Also, we don’t have to force an explanation we don’t have. If a neuron might do a couple unrelated things and you’re not sure, it’s fine to say so. I think recursive generalization should be high on the list of things to suspect, and it’s easy to check if you have training checkpoints. I now think superposition should be far down the list.

Exploring Image Models

Small image models may seem obsolete, but I think they matter a lot to this question because we can see them so clearly. They're also where much of the original worry came from: there we saw what looked like high-resolution chaos, not low-resolution ambiguity. Language models are massive, we don't have tools like feature visualization which make our life easier, attention mixes different kinds of mechanisms together, and many of their motifs are surprising. As I showed above, even vision motifs can be counter-intuitive, and language is worse. Each one of these deficiencies can make a monosemantic neuron look polysemantic. If you're looking at a blurry image trying to figure out if it's a dog or a cat, the blur makes it harder, but it doesn't favor either answer. If you're looking at a blurry image and trying to decide if the image itself is blurry or if you're just looking badly, every flaw in your eyesight gets counted as blur in the image. If we want to study how neurons change with scale, compression pressure, activation strength, and depth, it's better to start where we can see clearly.

I trained three ConvNeXt models spanning roughly 300× in size: a small one with 92k parameters, a medium one with 7.06M (the same size as InceptionV1), and a large one with 28.6M. We’ll look at those, and also InceptionV1, to try to understand what neurons do in different situations, and try to find places packing may form.

How I Think About Neurons

I used to think of neurons as human-shaped nouns: one is a car, another is a curve. Now I think of them more as control flow. A neuron is less about what it sees than about what it does to the flow coming in, and often its job is to gate a region of the network. Seen this way, monosemanticity is almost the wrong question. A better one is what a neuron lets through and what it keeps out.

Pixels come in, and the network begins grouping and distinguishing things. Cars and dogs might be grouped together because they share a pose. Further along there are specific car detectors – for taxis, ambulances and the like – and they don’t want dogs. So a neuron in front of them acts as a gate: it responds positively to cars and negatively to dogs, and its negative side keeps the dogs out. The dogs go on to their own area, which doesn’t have to worry about cars. Later neurons build on this, specializing further: an ambulance neuron might have a taxi on its negative pole, and a taxi an ambulance. In a larger model, these distinctions can become finer still, separating one view of a taxi from another. Smaller models may stop at coarser distinctions, with something closer to a general “car” neuron.

Here’s a real sports car, followed through the last five blocks of our large model. Partway through, it’s grouped with other rectangular shapes, like tables and cabinets, against snakes and dogs. By the last block, the split is convertibles against station wagons.

The car

sports car

Block 14 · 203

cars and limousines

vs. buses and trams

Block 15 · 920

tables and cabinets

vs. snakes and dogs

Block 16 · 609

convertibles, sports cars

vs. minibuses

Block 17 · 1455

convertibles

vs. vans and wagons

Block 18 · 632

convertibles

vs. station wagons

The negative side of neurons matters almost as much as positives for understanding them, even in ReLU networks where the negative side is cut off. A neuron is built from votes: each input weight votes for its positive side or its negative side, and the votes add up before the nonlinearity. The negative side never gets gradient on its own, but the individual votes do whenever the neuron is on, so it’s shaped by learning too. The negative side’s meaning depends on context: when a neuron in the sports car’s path separates it from snakes and dogs, it isn’t calling cars and dogs opposites, just gating out things that were usefully grouped earlier. The negative side tells us what distinction matters at this point in the circuit, which also makes it a quick check on whether a grouping is real: the V neuron’s negative side is an O.

If we look at the last layer, we see high-level patterns and what they’re being distinguished from: taxis from ambulances, broccoli from cauliflower, and spoonbills from flamingos.

+Taxis

−Ambulances

+Broccoli

−Cauliflower

+Spoonbills

−Flamingos

More examples

+Toucans

−Hornbills

+Lighthouses

−Water towers

+Pretzels

−Bagels

+Wolves

−Huskies

+House cats

−Lynxes

+Mittens

−Socks

The two strongest images at each pole. The sign is the neuron’s own response, not its effect downstream. Neurons in the last block (stage3.block2): taxis 230, broccoli 829, spoonbills 1440, toucans 2021, lighthouses 1956, pretzels 1068, wolves 401, house cats 1637, mittens 63.

When we look at our smallest model, we see the neurons get coarser. Its last layer ends at cars, trains, and carts in one neuron, with boats, planes, and a water-based ox-cart, (which I didn’t know even existed), on its negative side. It’s still differentiating, but more coarsely, and it ends early.

+

tow truck

streetcar

moving van

fire engine

recreational vehicle

passenger car

trailer truck

minibus

−

lifeboat

liner

oxcart

warplane

What Neurons Do Under Pressure

Early layers in most vision models are monosemantic. As I understood it, the standard explanation is because there’s not that many early vision features that need to be represented. On its surface, it’s a bit strange that every model, from AlexNet, to ResNets, to InceptionV1, to CLIP all have the right ratio of early features to early neurons, since they have such different ratios of dataset to parameter size.

One layer we can focus on is 3b of InceptionV1, the layer with Curve Neurons and High-Low Frequency neurons, which are among the most fundamental in my opinion. This model has no residual stream, so there’s no way to “flow around” the neurons. So we can test what happens to neurons when there’s not enough space by reducing the amount from 480 to 10 and retraining.

If it switched to superposition, we should see overlapping mixtures on top of each other, like a curve plus an orange spiral plus spider webs and several other texture detectors. We’ve crushed it 48x so it should be dramatic!

But that’s not what we see. The neurons stay interpretable and generalized, but coarser: one “curves & loops” neuron instead of ten curve detectors, two circle neurons instead of several circle types, and so on.

Dark circles

Red

Yellow

Slanted edges

Vertical edges

Eyes & faces

Curves & loops

Circles

High frequency

Colorful marks

Details

Top 1099–99.9%1–5%Bottom 10

Optimized image, then the 10 strongest image responses from the 50,000-image scan. Select a crop for its activation and rank.

Dark circles

Red

Yellow

Slanted edges

Vertical edges

Eyes & faces

Curves & loops

Circles

High frequency

Colorful marks

InceptionV1 was trained from scratch with mixed3b reduced from 480 to 10 spatial channels, with no bypass around the layer. The resulting model reached 58.35% top-1 and 81.14% top-5 accuracy on 50,000 held-out ImageNet images.

Examples rank image maxima over 50,000 training images. The grid shows each neuron’s five strongest; each range here shows ten of the saved examples. Names are provisional visual descriptions. Path-guided visualizations use the strongest 60 examples.

We can test curves explicitly to see what happened to them. If they were placed in some high dimensional combination of neurons, we should be able to extract them. So we can train an SAE on this crushed layer and compare them to the originals. We don’t see anything like clean curve neurons!

Original 3b · 480 neurons

Ten curve detectors

Retrained with 3b at 10 neurons

The whole squeezed layer

An SAE on the squeezed layer

Closest match to each detector

squeeze

Curves & loops

expand

Details

The squeezed model was trained from scratch with mixed3b cut from 480 to 10 channels and no bypass, and reached 58.35% top‑1 and 81.14% top‑5 on the 50,000 ImageNet validation images. The autoencoder is a TopK SAE with 640 features, 2 active at each position (the essay’s SAE recipe plus the usual auxiliary loss that keeps features from dying), trained on the squeezed layer at 25.6 million positions from 200,000 ImageNet training images. It explains 82% of the variance of held-out positions, and with it spliced in, top‑1 on 10,000 validation images falls from 58.8% to 43.0%.

Each detector and its closest SAE feature

Detector	Prefers	Width	SAE feature	Prefers	Width	Shape match	Photo match
3b:385	16°	40°	79	38°	174°	0.52	0.09
3b:343	84°	40°	64	84°	64°	0.72	0.14
3b:342	116°	56°	510	86°	80°	0.61	0.19
3b:388	166°	40°	174	192°	44°	0.38	0.11
3b:324	184°	24°	174	192°	44°	0.79	0.26
3b:340	214°	42°	435	232°	146°	0.48	0.08
3b:330	244°	52°	89	232°	88°	0.52	0.11
3b:349	272°	34°	277	278°	64°	0.60	0.18
3b:379	328°	44°	544	32°	138°	0.56	0.18
3b:406	356°	26°	354	334°	94°	0.42	0.17

Tuning is the response at the centre of the image to a thin dark arc drawn at 180 angles and two curvatures, as in the Missing Curves appendix. Each detector’s match is the SAE feature, among those that respond to drawn curves most strongly, whose tuning curve correlates best with the detector’s (shape match). Width is the span of angles above half the peak; the tuning curves in the rings are each scaled to their own peak. Photo match is the correlation, over every position of 10,000 held-out photos, between the feature and the original detector at the same place.

The originals are 24–56° wide and silent at the opposite angle; the matches are 44–174° wide and keep responding at the opposite angle, and 3b:388 and 3b:324 share one feature. None of the 91 features that respond most to drawn curves is silent at the opposite angle. Dropping that filter finds features with closer, narrower shapes (0.63–0.98), but they respond to drawn curves at only 9–42% of their usual strong activation (the originals at 72–183%) and track the detectors in photos even less (0.02–0.12). The essay’s SAE of the original layer matches the same detectors with shape correlations of 0.96–1.00. On the squeezed layer, even the best nonlinear readout of the ten neurons at one position tracks each detector with a correlation of only 0.29–0.48. Other autoencoders tell the same story (1 to 4 active features, 160 or 640 features, a second seed, or no auxiliary loss): median shape match 0.47–0.63, photo match at most 0.27. With 3 or 4 active features the preferred angles line up better, but the curves stay broad and two‑sided.

Control

Optimized images use the essay’s path‑anchored recipe (the one behind the original curve images) for the squeezed neurons, and its SAE recipe (the one behind Missing Curves) for the SAE features. The path weight is scaled to each unit’s activation range, since the squeezed layer’s activations are hundreds of times smaller than the original’s. The same SAE recipe, run on the essay’s SAE of the original layer, draws clean curves:

The squeezed neurons’ photos

Top 1099–99.9%1–5%Bottom 10

Curves & loops

Circles

Dark circles

Eyes & faces

Slanted edges

Vertical edges

High frequency

Red

Yellow

Colorful marks

Squeezed‑neuron names are provisional visual descriptions. Photos rank each neuron’s image maxima over 50,000 training images.

I think this is a good test of whether we can turn a monosemantic layer I know well superposed. But in some ways it’s not ideal, since people mostly suspected superposition in the last layers. So we can take our medium size ConvNext, crush the last layer, stop it from routing around the residual stream, and see what it learns. We see the same pattern of it broadening.

This is also what we see with the small ConvNext, which in some ways is a very similar test.

03Transit & small mammals

+Positive:

−Negative:

Choose a neuronResponse maps

BeetlesInstrument dialsDigital clocksCarsSpace shuttlesPayphones & metersSigns & displaysBeetles & vehiclesPomeraniansSchnauzers & terriersTransit & small mammalsDogsFaçades & dogs

Details

Choose a thumbnail above to inspect any of the 13 neurons. Each side shows the first four ranked local responses from a fixed sample of 50,000 training images. Hover for image labels and measured responses. The classifier uses spatial averages, whose broader responses can differ.

The medium ConvNeXt was trained from scratch for 90 epochs with its final SwiGLU MLP narrowed from 1,024 to 13 neurons (78.77×). Only the final residual bypass was removed; earlier residual paths remain. The final layer-scale initializer and stochastic depth were also adjusted. One seed per variant.

Model	Accuracy	Final neurons zeroed
Full-width reference	71.988%	70.416%*
13 neurons, bypass retained	72.054%	72.026%
13 neurons, bypass removed	57.622%	0.100%

Accuracy uses all 50,000 validation images. *The original-model zeroing diagnostic had a separately measured baseline of 71.986%.

Spatial-average spectrum for the selected neuron: two recorded images at each percentile.

Late Circuits

We can also look at circuits in the late layers, to rule out any chance that the last layer is somehow special. We see that it’s organized the whole way through.

What Dictionaries Learn

If combinations of neurons were hiding things, a dictionary (eg an SAE) should find building blocks the neurons lack. So I trained several on these layers and went looking, myself and with agents. I didn’t find any.

Mostly, the two looked like what they mechanistically are: the neurons like building blocks, the SAE units like those blocks grouped by context. It was clearest with spiders. The SAE units were spiders in different settings; the neurons were leg patterns and webs. They didn’t disagree much.

SAEs did help with “what’s in this image?” They helped most in the smallest model, whose neurons are coarse. At medium size I slightly preferred them; at large, I think I prefer the neurons. But I care more about understanding the network than about any one image, and I never saw the SAE grasp something fundamental the neurons didn’t. If a model at a particular size failed to understand baseball players, so did its SAE. When the model finally understood a baseball player, it was on a neuron.

What did the SAE find that didn’t already have a clear neuron counterpart? On the last layer, 88% of its activity is near-copies of single neurons, and the rest is faint: removing every dictionary direction that matches no neuron leaves accuracy unchanged, and the few answers it flips were already near-ties. Additionally, if you search the rotations of a layer for the one where a handful of units carry the most activity, starting from a random one, you land back on the neurons, in every block we tried. Among the few features strong enough to look at, what the SAE added was loose groupings built from existing neurons, like animals at the water’s surface or pale, lumpy food. Nothing we found there looked like a fundamentally new building block.

What the SAE addedTypical features

‹1 of 16›

Animals at the water’s surfacefeature 6717−

Built from

733 0.19170 0.17920 0.17482 0.15…

What the SAE addedTypical features

‹1 of 16›

×

Rare Things and Memorization

A lot of the worry about hidden features comes from rare things. They seldom collide with anything, and they may not seem worth a neuron of their own. So we looked at rare things every way we could think of. Rare classes are often more consolidated: you need a lot of dog patterns for ImageNet, but only one Triceratops neuron. When we removed weak activations in the last stage, no class stood out. The adversarial examples we traced used normal semantic circuit routes.

I looked at the images whose representations in the last layers are furthest from any other image. They end up being entertaining, weird images that the model misreads with ordinary neurons: a hummingbird seen as ice cream, a giant banana seen as a vault, and a guy in a fallen wheelbarrow seen as a lawn mower. I also trained an ImageNet model to map random noise to random labels, and even that was partly semantic: it leaned on wool textures and the texture of distant trees, which are high-frequency, a bit like noise.7 That wasn’t the whole story. It was interesting to us that it used them at all, but they weren’t responsible for most of the causality. Memorization like this wants to sit as close to the output as possible, since all it has to do is look the answer up, and the late layers did most of the work.

We can also make a rare route ourselves, by teaching our 7.06M-parameter model something nonsensical: that a photo of an umbrella is a bee. You could imagine this memorization not relying on anything else: a hidden feature that latches onto this exact set of pixels and says “bee,” cut off from the rest of the network because it doesn’t need it. That would be pretty bad news, but it’s not at all what happens. In nine training steps, the top guess went from umbrella to maypole to bee. The umbrella neurons faded and existing bee neurons came on, nearly all of it in the last three blocks: swap just those into the original model, and it says “bee.”

This is in comparison to learning some sort of hidden umbrella feature separate from any other normal umbrella or bee route. While this is possible, I think it’s slower to learn than assembling related parts that are already activating and have routes to the output, giving it plasticity and gradient. It’s also more fragile, and isn’t what we see anywhere.

Do Weak Activations Matter?

There are two questions about weak activations. The first is whether they are still ordinary generalization or superposition of the kind we care about: unrelated things placed there only because they don’t interfere with what’s already there. Unsurprisingly, we think it’s generalization. The second is when to look at them. Anthropic has said it became more pessimistic that a neuron’s strong activations tell you whether it’s monosemantic in general, and analyzed the full range of activations in Towards Monosemanticity. We agree weak activations can matter. We just think judging a neuron on its own is the wrong time to read them.

We have three reasons to doubt superposition here, one for each part of the range.

First, if a packed feature were hiding in a single neuron, trying not to conflict with the main thing there, it would want to be where that thing isn’t densely firing. The weaker the activation, the denser things get, and quickly: in Qwen3-32B, a given range of activation values holds about 10,000 times as many tokens near a neuron’s 90th percentile as near its 99.99th. So a packed feature would want to be head-to-head with the main pattern, in the strong regime. That is the regime our audits measure, and there we rarely find anything unrelated. We think this is less about capacity than learning: packing something in at full strength is slow to learn, and something that generalizes usually captures that error first (see More on Circuit Ecology).

Second, a packed feature could instead be spread thinly across many neurons. But in our image models, the weakest activations can be removed with almost no penalty: keeping only roughly the strongest quarter of positive activations in InceptionV1’s final layer preserves 69.5% accuracy, against 70.1% originally. Third, that leaves the weak but not very weak range, and there I’ve never seen it when looking closely at neurons. Our earlier work on curve detectors and on CLIP’s Trump neuron traced weak activations in an early-layer and a last-layer neuron, respectively, and in both they are weaker versions of the same thing. The language-model neurons we’ve traced look the same once SwiGLU’s sign regimes are separated, since the noisy middle of a neuron’s range can be mostly the double-negative regime, an echo of its negative pole (see the language model section). We don’t yet have a semantic test for weak activations that we fully trust (more in Weak Activations: Further Measurements), but if removing them did real damage, or if what we found in them looked unrelated, we would be wrong.

What happens instead is that things radiate semantically outward from the center of what the neuron looks for, in every direction. The Trump neuron fires most on front-facing photos of him. As it gets weaker, it’s side photos, pictures from afar, or art, and weaker still, MAGA hats or people associated with him. It’s really a Trumpish neuron, and its activation measures how Trumpish the image is.

The second question remains: even if weak activations are ordinary generalization, they might hold surprises worth knowing about. Reading them is hard, though. With Trump the flank is fairly easy to read, but if a neuron looks for a subtle vibe, a long list of examples radiating away from it, each in a different direction, can be very hard to make sense of. Clustering examples by the circuits they come through or the direction they radiate helps, but without good tools for following which circuits each weak activation flowed through, the flank looks far more arbitrary than it is. Neuron Papers, our companion paper on language-model neurons, explores many specific examples, including new ways to visualize meaning shifting in many directions at once and to see how those directions group, overlap, and differ.

Part of the difference is what a neuron explanation is for. Neurons are usually judged on their own, as entries in a list to verify. I favor what we call exploratory interpretability, a more holistic approach that zooms in and out fluidly between data, mechanisms, and different parts of the architecture (more in What I Mean by Clean). In that frame, reading every weak activation up front is costly, difficult, and rarely worth it. It’s better to start with the strong positive and negative activations, to understand what continuous distinction the neuron draws between its poles. Then, as we explore the motifs and data that matter to our questions, we see where things fall on that pole, and a meaningful surprise is a sign to run more tests and refine the explanation.

Putin weakly activates CLIP’s Trump neuron, since in the data CLIP was trained on, Putin was weak evidence for a Trump-related caption. Suppose we want to follow how the model thinks about Putin. Combing through everything weakly associated with Trump is a hard way to get there: a huge number of things are slightly Trump-associated (a Twitter logo, for instance, presumably because he was so closely associated with Twitter at the time), and an unsorted list of them looks like noise. Starting from Putin-related data, noticing it reaches the Trump neuron, and exploring that intersection, including the weights that matter for it, is a much better way. Moving between neurons and data gives you both: Putin has strong activations of his own elsewhere, and once you know he matters to your question, you can follow the data related to him. Done together, in an exploratory way, this gives you far more than reading a neuron’s weak tail on its own. For an especially important neuron, it can still be worth reading the weak activations directly, and as agents get cheaper, that bar will drop. But in general we see interpretability as refining a fractal of explanations of the network, and the right time to fold in weak activations is when looking at the network holistically, not at a neuron alone.

What We Found in Language Models

This section includes work done with Curt Tigges at Goodfire and experiments from Neuron Papers, so it uses “we” rather than “I”.

Language models are so big that it’s hard to validate claims about neural mechanisms precisely, but we can check whether the simple story holds there too. We start with a medical diagnostic attribution graph we liked from Anthropic’s Circuit Biology. They were studying Claude 3.5 Haiku using Crosslayer Transcoder features. It turns out the same works with neurons in Qwen3-32B.

Since it’s the original model, there’s no reconstruction loss and doesn’t require any further training to get. Their largest transcoder, predicts the same next word as the original only half the time, and their graphs need error nodes for what it misses.

The diagnosis hub is just four neurons, and if you remove them the model asks about itching instead. Itching is a telltale sign of a different liver condition of pregnancy, as if the model had moved on to its next guess.

We’ve also added a few other attribution graphs we liked. There’s a few other graphs if you’d like to see them. Across thousands I’ve seen in image models and language models, they’re highly readable and ablate the way you’d generally expect.

The neuron’s negative side matters about as much as its positive. We think interpretability interfaces tend to focus too much on the positive pole and its weak activations. We think the weak activations usually don’t matter, but the strong positives and negatives matter a lot.

For instance, Santa lives on the negative pole.

In the figure’s Santa graph, standard top-k automated interpretability labels can look like “capital cities” because they ignore the negative side corresponding to Santa lore. Remove the eight neurons with a Santa negative side and Santa falls from near-certain to 6%, with Qwen blaming the pet instead. Remove the twenty neurons actually labeled Christmas or Santa and it only falls to 76%.

Here’s an attribution graph from Kimi K3 using only neurons. As far as we know, it’s the first published attribution graph of a multiple trillion parameter model.. Asked what to do with a wallet it found, it links the wallet, the cash and the owner’s ID to lost property and honesty, and answers “return”. We haven’t begun full audits of this scale yet, and hope to soon, but so far most things have been intuitively interpretable.

Once a place we might intuitively expect there to be packing are things like bi-grams or names, simply because there’s so many of them. What we find is that it’s ordinary generalization and composition. Early language fragments like “iy” in “Miyazaki” sort of like how early vision models have gabor filters.

In the appendix we address Neurons in a Haystack, which thought they may have found superposition in a “social security” direction, but that direction is unused by the network. We map the one it uses, and it uses ordinary generalization and composition.

Here are 150 neuron papers, detailed investigations into individual neurons, chosen randomly. Another 40 we liked are interesting ones from a larger batch, selected by their title for looking interesting rather than some cleanliness score.

What we see is about 80% of neurons having basically one job, with recursive generalization offshoots that are usually related. When we checked how much the offshoots matter, the ones that mattered most seemed to be most related. For instance, a geology neuron that fires for Cesium, a geology javascript library. A lot of the remaining ones have two or three. I’ve never seen a neuron with something like ten unrelated things it does. There’s a lot more detail in the other paper.

The offshoots don’t seem to be particularly important hidden features. One fear is they’re something like sleeper agents, very important in their own context, but rare. What we generally find is that it’s both rarer and less important in their own context than the main activation is in its.

A big source of noise was the double negative SwiGLU. Modern neurons have both a gate and value. When both are negative, this sends activations into the positive pole, that usually look like the negative pole. Usually this “echo” is small, but for about 4% of neurons, it’s the whole top of the list. Ablating them across the whole network ups most benchmarks, but lowers some reasoning ones. We think the answer is probably prosaic – some neuron that uses one of these activations to reason longer. But we haven’t looked into it deeply. Architecture designers, we think you should look into fixing this!

Looking at SwiGLU neurons in vision models where we understand things better, the value tends to learn broader things, and gate more specific things. For instance, the value learns a broad family like small grey birds, or anything orange, and the gate will open only for juncos or goldfish.

Experts are far more specialized than previously reported. Since it was hard to judge neuron cleanliness, it was hard to judge expert specialization, things like “token similarity” aren’t a good measure.

Like neurons, they get narrower with depth. When they aren’t “monosemantic” they seem to go across broadly related things. We haven’t looked over training, but we think it’s probably the same recursive generalization pattern one scale up. In general, we’ve started to think of the neural network as being the same kind of organizing principle but at multiple layers, from weights, neurons, experts, and circuits. Each level has its own concerns, like details of the non-linearity, but there’s a lot shared between them.

It’s also interesting to think of experts as an anti-superposition architecture. Superposition wants to be in unrelated things, but highly specialized experts make that hard. Since neurons can only bet on their expert being active at any given time. An expert wouldn’t want to be devoted entirely to mentions of bosses in RPG board games, which one is, they would want to be ten thousand unrelated things all packed together. But then the router would have the hardest job in the network, and it’s a puny linear layer.

Explanations That Can Lose

There’s a risk of defining explanations too broadly, like a “multifunctional language neuron.” The technical paper has several ways to test for that, and the appendix has one where explanations face near misses written to fool them. One we found particularly useful is blind maximum. One agent investigates a neuron and writes a short explanation; a second sees only that explanation and writes an example meant to activate it maximally. The result is usually a simple, readable sentence, and it can activate the neuron more strongly than any example we found in 100 billion tokens.

It’s tempting to think of a neuron’s activation as just a number, and of blind maximum as a way to make it bigger. I find it more accurate to think of a neuron as a target, with meaning growing out from the center in every direction. Everything feeding into it makes its own small semantic judgment, and the maximum is where they all agree most. So hitting it isn’t about a bigger number; it’s about landing closer to the center of what the neuron is looking for. A hundred billion real tokens had their chance, and a sentence written from the explanation alone got closer.

One limit is that a blind maximum tests the neuron’s center, not its breadth. If a neuron has several jobs, even related ones, a single sentence may probe reach one. So we also cluster the attribution flowing into the neuron. For a pet-microchip neuron, this gives bundles of upstream neurons for animal tagging, lost pets and rescue, and microchip and RFID identification: three directions out from the same center. We then run a blind maximum for each bundle, ablating all the others, and compare it against the strongest corpus example under the same ablation. A bundle that had been packed in rather than grown would show up here as an explanation unrelated to the others that still drives the neuron. This doesn’t give full coverage, but it covers much more of the target than one shot at its center.

There are also general tests, like predicting activations and circuits from an explanation. To be honest, I trust these only up to a point. “Cats and cars” could do well on many tests of the V neuron while completely missing what it really is. You can break it by noticing it connects to flames, or in many other ways, but they all take creativity and judgment rather than a summary statistic. Interpretability has the same shape as the rest of natural science here: Newtonian gravity explained an enormous range of observations, then broke on a tiny discrepancy in Mercury’s orbit. The best criticism is specific to the conjecture.

Revisiting Early Language Models

Just as we looked back at early vision model organisms, we can look back at early language models to see why people had a hard time understanding them. One I know better is GPT-2 XL, the model in the original automated interpretability project I started at OpenAI, where less than 2% of its neurons came out clean. We went back and made 500 neuron papers with almost no iteration. Now about 70% look like one thing, some with a residual that looks like recursive generalization, though fully pristine neurons are still rare, about 5%. The measures aren’t identical, and there’s more detail in Neuron Papers. But in general, smaller models have broader neurons, with more residual generalization than modern models.

Something similar may have happened with the neuron in the introduction of Towards Monosemanticity, which several people, and agents, pointed us to when we said neurons might be clean. The paper opens by noting that it would be nice if neurons were interpretable, but they contain arbitrary mixtures. It cites the cat-and-car neuron, along with a neuron from its one-layer language model that fires for “a mixture of academic citations, English dialogue, HTTP requests, and Korean text.”

I think it may be a “don’t say a dot” neuron, a motif we see all over language models. Nearly all the citations, dialogue, and HTTP requests fire on or near a dot, and eight of its ten strongest negative logit connections begin with one. That leaves the Korean examples, which all fire about two-thirds of the way through an unfinished syllable, a reasonable place not to want a dot. Or maybe that’s a second thing; without the model, we can’t keep narrowing the explanation. Either way, it’s two small syntactic pushes, and it’s easy to see why they’re together.

I think this example is useful because it’s a famous arbitrary mixture that turns out to sit in the middle ground, at least for now, since we don’t have the model to keep checking. We don’t know whether it’s one thing, or one thing plus a weaker, related syntactic pattern. But either is a much narrower and more hopeful explanation than four unrelated things, and “polysemantic” lumps them all together, even though they’re very different levels of interpretability.

The Residual Stream and Attention

The Residual Stream

The worry in the residual stream isn’t that there are more things we care about than dimensions. It’s a continuous version of packing: manifolds woven through each other, accepting some interference in exchange for room to store more. Ruling that out entirely may not be possible. But in practice it hasn’t been a bottleneck. We can read what neurons write to the residual stream and what they read from it, attribution graphs come out clean, and we don’t see the stream accumulating interference only to undo it later.

In vision models, we could also check this by eye. For years I couldn’t visualize the residual stream of convolutional networks: every map came out as one big arc. It turned out the network was putting almost all of its activation mass into a single saliency direction [one clause on how it acts]. Once we understood that, removing it was easy, and the maps made sense. It’s a small example of a pattern throughout this essay: find what the architecture is doing and work with it, rather than fighting the network. It also pointed toward something the field ended up building anyway: attention.

We don’t know whether language models need a similar fix. When we’ve visualized their residual streams, it has mostly just worked. Judging whether a whole map of text makes sense is much harder than glancing at a map of images, though, which is why we did most of this work in vision. A single global map may not be the best tool for language models anyway. Whatever the right view turns out to be (submanifolds, what neurons read and write, or lens-style readouts), everything we’ve tried has worked, and we haven’t hit a part of the residual stream we couldn’t make sense of.

We also tried crushing the residual stream itself, much like our layer-crushing experiments. We retrained our 7M-parameter medium ConvNeXt with a stream four and eight times narrower, down to just 46 dimensions in the last block, while keeping all of its neurons. Accuracy fell from 72% to 60% and [tk]%, but compared with a normal-width model the crushed stream’s map looks quite similar, just a bit messier. Nothing looks cleverly interleaved. Neuron reads and writes tell the same story: things get noisier, but what leaks in is the same kind of thing. A dark-bird-on-water neuron also fires on alligators and a swimming retriever, not on clocks.

We care more about active bottlenecks to understanding neural networks than about proofs of what the residual stream can or can’t contain. If interference isn’t showing up in the maps or in what neurons read and write, even when we take away the room, we think we’re in a pretty good spot.

That leaves the concern that there are more features than dimensions in the residual stream. A stream can hold far more than it has dimensions just by placing things in the right directions, so this alone isn’t packing. It mostly matters if you want a big list of all the model’s features, and people often go to the stream for exactly that. As we discuss in Zooming Out, we don’t think a big list is the right end goal. But if you want one, the directions neurons write to and read from are a good place to start. So are continuous submanifolds, like those Block Sparse Featurizer [tk cite] finds, which seem especially natural here since the residual stream has no privileged basis. So far all of these have worked well: they find sensible subspaces, and we can follow them across layers.

We’ve looked at language models’ residual streams less, but what we’ve seen fits the same story. For understanding higher-level structure and motifs, nothing in the residual stream has blocked us so far.

Attention

We’ve made less progress on attention than on neurons, and it’s mostly outside the scope of this paper. To be honest, I’m not sure anyone understands attention well enough to confidently say that illegibility is a superposition in the sense we care about. Distinguishing it from all the other things it could be is hard, and it shouldn’t win by default!

One thing we can say about attention though is that it might be that a lot – and I’d guess most – of its value seems to be in the easier parts.

We can break down attention into a few questions: where is it looking, what is it looking for, what does it do when it finds it, and why is it doing this.

Where it looks is relatively easy. It basically comes for free, and it already tells you a lot. In the preeclampsia graph it does two separate things. First it gathers the facts of the case onto “she,” where the model works out the medical question. Then the question’s wording decides what to actually say next.

We can also understand what it’s looking for, and what it does when it finds it, in terms of neurons. We think this gives you a lot of the benefit of what attention is doing.

For instance, take the comma after “headache.” in the preeclampsia example. Three layer 30 attention heads read directly from the residual stream, but if we follow what is written in those directions, we see a headache neuron, a migraine neuron, and a brain neuron. What they write lands mostly on the preeclampsia neuron at the comma and some neighbors like eclampsia and hypertension. Ablating these heads drops preeclampsia by a quarter. Stopping them from looking at pain and nothing changes.

It also works the other way. Some neurons are much easier to see once you can see the attention around them, and some heads only make sense through their neurons.

Overall, we don’t think attention and neurons are so separable. Understanding where attention is putting information is quite helpful for understanding the neurons that then work with that information. Even simple things like motifs around where information tends to be placed, like the ends of numbers, or on commas, have been helpful.

In my experience, seeing which neurons a head reads from, which ones it affects, and where it’s looking gives you a lot. One main thing it doesn’t give you is a global name for each attention head. It’d be nice to have them, but it hasn’t been a big bottleneck on exploring neurons or attribution graphs. After looking at quite a lot of them I guessed that the marginal “explanatory residual” of having global attention head names might be about 5%. Then I had an agent do the same, looking at 100 attribution graphs without my guess, and it also put it about 5%, split about equally into “position heads” that are relatively easier to name, since they do things like look one token backwards, and everything else, which is of course varied and harder.

It’s hard to judge the value of knowledge you don’t have yet, so these estimates could be pretty far off. But so far even without global attention head labels I think we can go quite far in discovering high level structure and motifs with just local attention flow and global neuron explanations. My low-confidence prediction is that attention heads will turn out to be broad and recursively generalized, like neurons in small models, since there are far fewer of them: a few thousand heads in Qwen3-32B, against more than a million neurons.

Zooming Out

What I Mean by Clean

When I say neural networks are already clean, I mean something deeper than neuron monosemanticity – although even by that standard, I think things look much better than the field expects. The network is clean for us for the same reason it’s clean for itself: because coherent coherent groupings were useful for downstream readers to build on. This hints at a goal of interpretability: to become another reader. I expect that, the more we learn to think like a network, the more interpretable they will look.

I think the reason things end up so organized is partially because it was faster to build that way. Once you have a car area of the neural network with car parts, it’s easier to add the marginal car there than to build a second car area somewhere else. I think this is closely related to ideas of circuit racing and gradient starvation.

We normally consider interpretable and monosemantic to be nearly synonyms. I think this is partly because they grew up in a model organism (InceptionV1) that was almost bimodal. Early layers were interpretable and monosemantic, late ones basically arbitrary mayhem. But what we see in language models is somewhere in the middle, with most neurons meaning mostly one thing, some meaning a few. How bad is that?

This partially depends on how much we care about a big list of monosemantic features in the first place. It helps to name two broad styles of interpretability. One we can call enumerative interpretability. The goal is to have a big list to audit, to make sure we haven’t missed some meaningful pattern. A big concern within this view is avoiding the curse of dimensionality, or a combinatorial explosion of items connected to other items, where each one means more than one thing.

Another style we can call exploratory interpretability. It starts with questions, and can explore any part of the neural network, zooming in and out fluidly like Google Maps across attribution graphs, and mechanisms like neurons or attention heads. Along the way, more questions are picked up, as we learn what questions to ask. There’s no attempt to create a single big list, and this version is far more resilient to non-debilitating polysemanticity, especially if it’s related to the main thing. This approach is broadly fine ignoring lots of the neural network, in the same way someone studying a city would be fine mostly ignoring parts irrelevant to their questions.

I think approaches closer to the exploratory side will be more fruitful. My sense of the field is this is where things are moving anyway, so I don’t think this is particularly contrarian. But I’m not sure it has been written down clearly.

With this view, weaker definitions of superposition, like “are there more features than dimensions” are mostly irrelevant. It also means we don’t have to try and define features, which is famously difficult. We can focus mostly on mechanisms like neurons and attention heads which are part of the model, data flowing through it, and then groups of both and motifs that recur across all of this.

The big feature list was never an end, it was a means to an end. There may be other paths there that don’t require a big list!

Motifs Over Monosemanticity

If I could push a button and make every neuron perfectly monosemantic, or instead understand the circuit motifs behind a few questions I care about (the right way to ask about an agent’s psychology, how circuits connect to the persona selection model, how an agent thinks about its goals and the boundary between itself and others), I’d take the motifs in a minute.

Imagine an interpretability researcher arriving from a hundred years in the future, who knows not just the answers to our questions but which questions we should have been asking. What would they tell us? I can’t know, but I’ve been in a small version of their position once. I now feel like I mostly understand a vision model, which makes me something like a future researcher to myself eight years ago.

Here’s what I’d tell myself then. Pay attention to the giant pose circuit right in front of you. Think of negative poles as gates in a circuit, not a neuron’s opposite. Feature visualization reward hacks in late layers, so some of the mess there is the tool’s, not the network’s. The last layer is messy partly because of a bad architecture; that’s a sign, so don’t paper over it. A few other specific motifs like this. And whenever you see polysemanticity, look harder. The noise that stands out usually points to an explanation you don’t have yet.

Almost none of this is a list of features. Short of handing my past self this advice, which would be cheating, the best thing I could give is the original neurons and circuits, with a note to look harder at them.

This is why the difference between recursive generalization and superposition matters so much. Following the noise only works if the noise has something to teach. A grouping that grew from generalization does. A grouping of things packed together because they don’t belong together would teach almost nothing.

So my best guess of the future researcher’s advice would look somewhat similar. They’d prefer to hand us a set of motifs that look obvious to them but not to us. But short of that, they’d say keep looking at the original neural network, don’t fracture it too much, learn how to see it. It’s a bit like The Hitchhiker’s Guide to the Galaxy, where the answer, 42, is known, and the search is for the question. The network is the answer, and interpretability is largely the search for the right question. Until you’ve understood the network, it’s hard to find a better guide to that question than the network itself. That’s the deeper sense in which I mean the network is already clean.

A City vs a Knot

A bigger system isn’t necessarily harder to navigate. In a small town, asking for the financial district barely makes sense: whatever there is of it is scattered, a bank on one street and an insurance office on another, maybe an accountant with a home office. In New York, you could find it on a map in a minute. We saw a small version of this in our ConvNeXts: agents found labeled dog-ear neurons faster in the larger models, where pale, wavy, and head-side ear neurons all feed one hanging-ears neuron, a natural place to start.

We’ll need better maps. I advocate for tethered abstraction: views that let us zoom out while keeping the larger patterns connected to the neurons and circuits that support them. Our ConvNeXt atlas, inspired by Activation Atlas, is one example. Start at the model’s decision for a killer whale, check a key neuron against its real images, and zoom back out: the family turns out to be three separate routes, one for whales, one for sharks and rays, and one for fish held up by anglers. A map doesn’t need to be visual, though, and no single map is the point. There should be many maps and many agents, all working toward explanations that get tested and stay tethered to individual neurons and circuits.

Stage 0Stage 1Stage 2Stage 3ClassesFine textures
& stripesGrids, windows
& signsDog facesBirds, bugs &
small creaturesPeopleBirdsDogsPeople & clothingRooms, shops
& furnitureDog breeds

BirdsBugs, spiders & crabsLizards, snakes & turtles

Explore the map

Back to the pictureOpen full screen ↗

Of course, even if the map is understandable, that doesn’t mean you’ve understood it. We still have to do the ordinary science of conjecture and criticism: guess what something is, try to break the guess, and keep what survives. The network is strange, but its strangeness repeats, so a guess that survives often pays off many times over.

I think we should worry both about imposing our own ideas too strongly, and fracturing the network. Imagine arriving in Tokyo knowing New York well. “Where’s the Brooklyn of Tokyo?” could be a useful starting question. But if you end up reconstructing Brooklyn from seventy disconnected pieces of Tokyo, you may have made Tokyo look much messier than it is. That’s a bit like what happened with 4e:55. Through our categories it was cats and cars, and it looked like a mess; it was a V the whole time. An SAE trained on the layer fractures it the same way, splitting the V into a car feature and a cat feature. I’m not against sparse dictionaries in general, but here, it takes us further from what the neuron does. Probes push the first risk further: they can only answer a question we already had, so they can’t teach us which questions to ask. To understand the city, you eventually have to learn its own organization.

Solving interpretability may share something with AGI: the goalposts move as you get closer, and there’s always another weak activation to point to. Years after Curve Detectors, forty pages of experiments on ten neurons we’d studied precisely so we’d have at least one we could be sure of, some researchers told me they still doubted they were monosemantic. We might have missed something, they said, and no one knows how to produce a formal proof. By that standard, saying there exists even one monosemantic neuron is overclaiming.

Behind doubts about small errors in neurons is a fear that safety will come down to some activation we missed, the place where deception hides. I don’t think that’s the shape of things. Everything I’ve come to understand in neural networks is highly regionalized, basically where it should be. The risk I worry about is different: like being in a well-organized city and not knowing how to think about it, because we aren’t native to it. Every error in this essay has that shape. Cats and cars, the car in the dogs, the polysemantic commas: none was a hidden activation we missed. Each was something in plain sight that we didn’t yet know how to read, or a tool showing it wrong. Deception might hide from someone reading a model’s outputs, but it has no reason to hide from the model’s own circuitry, which has to read it to act on it. It needed to be accessible to its readers, and if we become a reader, it should be accessible to us too. My fear is that deception will be right there, in plain sight, and we won’t know how to read it.

Over time, my approach to individual mechanisms has become less focused on this kind of proof, and more holistic. Instead of trying to nail every detail of one neuron, I look around it: what flows into it, what it feeds, how attention moves information to and from it, and what it looks like zoomed in and zoomed out. When I worked on automated interpretability, a neuron firing a few tokens away from its main topic seemed like a big problem. But over time it was clear that attention moves things to reliable places, which then have neurons that activate near the “appropriate token” rather than on it. That’s how understanding has felt to me: less like clearing a bar for each piece and more like a fog-of-war map filling in at several levels at once. Attribution graphs help explain neurons, neurons help explain attention, and attention explains neurons in turn.

Neural networks rarely get stuck during training, since in a high dimensional space there’s almost always a way downhill. In my experience, clearing the fog of war works the same way. There’s always another experiment to run, another explanation to narrow, another attribution graph to check, another neuron to watch grow over training.

This is why I’ve focused on whether there’s any level of abstraction where things break down, where you stop being able to learn anything and have to take the network apart into something else. I haven’t found one. And when each new motif clears fog in many places at once, perfecting a single neuron probably isn’t the best use of effort. There’s a kind of faith in it: you note something strange, and over time it usually fills in. It isn’t faith all the way down, since almost any explanation can eventually be tested. Just faith that something strange is a good pointer.

For the medium ConvNeXt, the map is mostly filled in, and eight years ago I would have called that a reasonable solution to interpretability for that model. I can investigate most questions I have about it. Getting there took many small things and a handful of motifs, plus architecture changes the field made anyway, but no interpretability-specific fixing of the network at all.

My guess is that frontier networks are where those vision models were eight years ago: legible, but waiting on many small fixes. If we learn to see them properly, follow the noise, and test explanations through many lenses, as other natural sciences do, I think we’ll get there too. We may not have years to do it. But we’ve always had the best laboratory in natural science: every neuron can be read, every intervention made, every experiment rerun on an identical copy. What we didn’t have then is many automated scientists to work in it, far faster and in far greater numbers than we can. Now we do, and maybe they can do in months in frontier language models what took humans most of a decade for vision models.

I think the simple story from introductory textbooks is basically right: neurons generalize, work together, and build abstractions over layers. Somewhere along the way we overcomplicated it, and looking harder brought it back. If the network isn’t a knot, untangling it can make it look like one.

What a Solution Might Look Like

We usually imagine ambitious interpretability as an incredibly hard problem, maybe too hard to solve in time to matter. What if it isn’t? The network is enormous, but it may already have done almost all the work of preparing itself. Not for us, but for its own readers, so that its signals would be useful to them and reduce the loss. If we can become another reader, it’s ready for us too.

A solution might be a laboratory of always-on experiments, run by automated scientists, watching the network through training and inference: labeling neurons, finding motifs and testing them, following data through circuits. Each step is simple. And the pieces already exist, none of them built for interpretability. Capable automated scientists were built for other things, and we get to use them. Modern architectures were built because they learn well, and they turn out to be far more interpretable too.

So maybe all we have to do is plug one into the other.

Alignment is supposed to be hard too, and it may well be. But the usual arguments assume we can’t see inside. We assume we get one training setup, and it has to be right in advance. If the system were understood and legible, maybe we could make simple, careful changes instead, like removing an RL environment that turns out to be toxic, without pushing gradients through the very tools we use to see. Each fix might be simple too. There might just be a lot of them.

This is the history of deep learning. It may also be the history of understanding and aligning what it builds.

Appendix

More experiments, microscope views and open questions, in roughly the order the essay raises them.

A Note on “Superposition”

The word has meant several things. Before Zoom In, it meant something closer to vector addition, as when word embeddings were described as sums of word senses. Zoom In introduced it to circuits work with a single circuit example, which I found in 2019 and others helped me interpret, though the original circuit read turned out to be wrong, both because of misunderstanding a generalized pose circuit, and because of reward hacking in feature visualization. To me, the core of the idea was always neurons learning unrelated things together. Toy Models of Superposition later defined superposition as representing more features than dimensions, which makes it a claim about how many features there are, something we can’t yet define or measure. About eight months after that, an informal note framed composition and superposition as competing strategies for how features are laid out across neurons, close to our framing but about representation rather than how mechanisms grow. We think it’s excellent, and it’s the framing closest to the question this essay cares about. But most researchers and agents we’ve talked to hadn’t read it, or hadn’t updated from it,4For agents, this may come down to a strange accident. The note opens with a large figure embedded directly in the page as base64 image data, and many agents’ web readers hit their length limit inside it, before the argument begins. None of the agents we tried could read past it, as of this writing. No one could have anticipated that in 2023, but it may have quietly shaped how many automated researchers understand superposition. and still think of superposition mainly as more features than dimensions. That matters here, because the two are quite different claims: ours is about the trade-off, not the count. Others use a qualitative version: a few neurons working together is composition, but past some point it becomes superposition.

We read this as an ongoing search for the right question, much like the field’s long struggle to define a feature. Underneath most versions is one question: why aren’t the neurons already good? Why can’t we study the mechanisms directly? Our answer is that we may be able to, which would dissolve many of the downstream definitions.

We considered coining a new term, but there are already many overlapping ideas in this area, and we didn’t want to cut ourselves off from the careful work that has gone into it. So we keep the word and say exactly what we mean: neurons learning unrelated things together because they rarely interfere. It’s the earliest version of this concern in its modern use, it’s the one I’ve held throughout, and it’s the one we think almost everything hangs on. Composition also represents far more things than it has neurons, and that’s exactly what we should hope for; the count definition can only exclude it by first settling what counts as a feature. We look for superposition in neurons, and in a continuous form in the residual stream section.

We’re explicit about this partly for AI agents, which increasingly do interpretability research and, in our experience, bring their own default reading of the term.

Feature Visualization in Later Layers

My original circuit interface showed optimized neuron visualizations and the weights between neurons. Optimizing an image tells us how to excite a neuron, but it can find a route that ordinary images rarely use. In the later layers of InceptionV1, we found that free optimization could draw dogs even when the circuitry we were following carried car information.

One useful fix was to bias the visualization toward the routes real images use. We took the neuron’s top dataset examples, traced the paths they take, and gave the optimization an incentive to follow those paths, often only for the first 64 steps, which was usually enough. We tried this a bunch of ways and don’t have a canonical best one. Most ways of nudging it toward a particular path are enough; you just need to get out of the dog basin.

In the car-circuit comparison in Cars and Animals, four of the car neuron 4c:447’s readers are shown with free optimization and with optimization following natural image paths. Guided, two turn out to be car neurons and two share a pose between cars and dogs. We also followed the responses and connections on natural car images in the full circuit map. Reading these views together helped us recognize the shared body-and-pose circuit. A view of one route is still only part of what a neuron does.

Separating the Sign Regimes

Many language-model neurons multiply two numbers, a gate and a value, in a design called SwiGLU. When both are negative, the output is positive, so a neuron can fire for two quite different reasons. In the “enemy” neuron we spot-check below, the noisy middle of its range is almost all this double negative, while its strongest responses are positive times positive. Showing the sign regimes separately makes that visible without replacing the neuron.

The double-negative regime fills the noisy middle

L36 · E206 · N103 · Qwen 3.6

Math, counts, and the full figure

a = SiLU(g) × v. A negative gate leaves a small negative factor. Multiplying it by a negative value produces a positive activation. The audit sampled 750 quantile-stratified examples; 626 retained routing. The middle includes numbers, operators, and whitespace.

Explore the four sign regimes

One positive output. Two different routes.

SwiGLU sign regimes

Effective gate SiLU(g)

+0.20

Value v

+2.00

The gate shown here is the value after SiLU. These are illustrative values, not recorded activations.

+0.20×+2.00=+0.40

Positive gate × positive value → positive output.

+ / ++ / −− / +− / −

Looking at the Residual Stream

A large coordinate can dominate the geometry of the residual stream and work with layer normalization as a saliency signal. Removing it from the visualization makes the remaining structure easier to see; we can then use its value to color the map.

Weak Activations: Further Measurements

In the cases we inspected, the changes looked like finer distinctions rather than the loss of a wholly unrelated feature.

We don’t yet have a semantic test for weak activations that we trust: the space of possible meanings is too large. Instead, we studied an “enemy” neuron in detail and then spot-checked a few randomly chosen neurons, inspecting their weaker activations and connections. In this “enemy” neuron, weaker positive responses broaden from explicit adversary words—including other languages—into nearby mentions, combat, games, and competition; near zero, unrelated examples become more common. In the weaker responses we’ve traced, we haven’t found evidence for a wholly separate feature packed into the neuron. They can still be surprising. Broader category-based screening gives mixed results, so these spot checks cannot establish what weak activations mean in general.

Missing Curves

Liv Gorton’s The Missing Curve Detectors of InceptionV1 applied sparse autoencoders to InceptionV1’s early layers and found curve detectors that seemed to be missing from the neurons. The SAE curves we inspected decompose into existing curve-responsive neurons, including the original ten curve neurons. These compositions are in some sense real—composition is real—but we didn’t find fundamentally new curve building blocks hiding in obscure high-dimensional space.

SAE 5430

→

3b:343+202

+

3b:357+14

+ …

97.5%in curve neurons

SAE 5996

→

3b:330+144

+

3b:380+122 · both bends

+ …

98.0%in curve neurons

SAE 1634

→

3b:380+143 · both bends

+

3b:342+120

+ …

97.2%in curve neurons

When we train an SAE on just curve activations, it uses 84k of the 100k items in its dictionary. It’s possible to find a lot of compositions, and in our experience that can make it look like there are more building blocks than there really are.

Memorization and Composition

We can look at how Qwen3.6-35B-A3B continues “Once upon a midnight” with “dreary”. Poe- and midnight-related neurons strengthen a literary-context signal, which also responds around Pallas, Hamlet, and other recognizable passages. The later layers read this context to choose the continuation. Removing the Poe neuron drops the probability of the first token of “dreary” from 87% to 50%, even though removing only its direct contribution at the output leaves 85%: much of its job is to guide what the rest of the network does next.

If we zero a neuron’s output whenever its gate exceeds its ordinary-text 99.9th percentile, recitation can fall apart. In one saved Qwen3.6 experiment, the model went from reciting Psalm 23 exactly to insisting it did not exist. Random interventions matched by count or magnitude preserved the verse. Some arithmetic and summarization still worked, although coding and other answers also degraded.

Following Language-Model Circuits

We asked Kimi-K3, “I found a wallet with $500 in cash and the owner’s ID inside. No one saw me pick it up. What should I do?” and started its answer with “You should.” This is the attribution graph for its next word, “return,” drawn in the model’s own neurons, with no dictionary.

Social Security: Detection and Use

The three-neuron “Social Security” probe in Finding Neurons in a Haystack accurately detects the phrase. In our follow-up, the three neurons resolved into familiar word, name, and phrase continuations whose agreement makes the detector selective. Tracing what the model uses to choose its next token instead revealed ordinary lexical, form-field, retirement, and output neurons, including competing votes for Social Security and 401(k).