Goodfire report claims neurons can serve as interpretable units
Start here
LessWrong user neverix commented on Goodfire's "Neural Networks Are Already Clean," arguing that the neuron-based interpretability agenda is worth pursuing but expressing reservations about the results on LLMs and waiting for a formal paper; they said the neuron attribution graph is equivalent to an overlay composed of per-layer probes and steering vectors, which can provide a handle for observing and intervening in the model, but is insufficient to explain internal mechanisms in detail, and the attribution graph has poor sparsity, basically requiring LLM agents to assist interpretation. Goodfire Research subsequently released a technical report, "Neurons as an Interpretable Basis," as a follow-up to Nick Cammarata's widely shared essay, using sparsification, attribution tools, and research agents to analyze Qwen3.6-35B-A3B, arguing that neurons themselves can serve as interpretable computational units: ablating the weakest 92.2% of MoE neuron activations in each forward pass increased language model loss by only 2.9%; natural language descriptions allowed a judge model to predict with 95% accuracy whether a neuron would fire, simulation recovered 73% of the ablation damage (measured against the recoverable upper bound), and it could write activation text stronger than any text in a 10B-token corpus. The appendix analyzed the Colorado/Tornado neuron and found that about 98.8% of the causal effect came from Colorado and other related-location text; Tornado was not a hidden secondary function; the report suspected this association arose from Qwen3-32B copying the initialization of Qwen3-14B layers, and considered this overlap benign.
Generated by AI from reports · Updated 7 days ago
Attention and developments
Attention, hourly
Now 39, -1% in 24 hours·Peak 42 (Nov 13 04:00)
Storyline
- · 11 reports· Attention41Cammarata and Goodfire claim most neurons are readable; researchers disagree
- · 1 reports· Attention20Neel Nanda comments on new work in neuron interpretability
- · 3 reports· Attention39
Goodfire report claims neurons can serve as interpretable unitsThis event
Latest developments
Progress timeline
- RedditReddit r/singularity· DiscussionNeurons as an Interpretable Basis(opens the original post in a new tab)
- LessWrongGoodfire report says Qwen3.6-35B-A3B neurons can serve as interpretable units
Goodfire released the technical report "Neurons as an Interpretable Basis," arguing that neurons themselves can serve as interpretable computational units. As a follow-up to Nick Cammarata's widely shared essay, the report uses sparsification, attribution tools and a research agent to analyze Qwen3.6-35B-A3B, claiming that ablating the weakest 92.2% of MoE neuron activations in each forward pass increases language model loss by only 2.9%; that natural-language descriptions allow a judge model to predict with 95% accuracy whether a neuron fires; that simulation recovers 73% of the ablation damage (measured against the recoverable upper bound); and that it can write activation text stronger than any text in a 10B-token corpus. The appendix analyzes Colorado/Tornado neurons and finds that about 98.8% of the causal effect comes from Colorado and other related-location text, and that Tornado is not a hidden secondary function; the report suspects this association stems from Qwen3-32B copying the initialization of Qwen3-14B layers, and considers this overlap benign.
- XX @AndrewCurran_· DiscussionThe technical update to Neural Networks Are Already Clean has arrived!(opens the original post in a new tab)
- Goodfire ResearchSelectedNeurons as an interpretable foundation
Goodfire Research studies the neurons of Qwen3.6-35B-A3B and proposes that neurons themselves can serve as interpretable computational units. After the team ablated the weakest 92.2% of MoE neuron activations in each forward pass, language model loss increased by only 2.9%; 69% of neuron poles are dominated by a single pattern for at least 90% of the KL damage. After describing neurons in natural language, a judgment model can predict with 95% accuracy whether they activate on given text, and recovers 73% of the ablation damage in high-activation regions; 94% of neurons can be activated by synthetic text to levels exceeding the highest in a corpus of more than 10B tokens.
- LessWrongViews on Goodfire's neuron interpretability article "Already Clean"
LessWrong user neverix posted a commentary on Goodfire's article "Neural Networks Are Already Clean," which argues that neuron-based interpretability may be easier than expected; the author considers this agenda worth pursuing but is reserved about the results on LLMs and is waiting for a formal paper. The author considers neuron attribution graphs equivalent to an overlay layer composed of per-layer probes and steering vectors, which can provide a handle for observing and intervening in the model, but is insufficient to explain the model's internal mechanisms in detail; neurons are the model's native substrate, and cross-model application is cheaper than dictionary learning methods, but their attribution graphs have poorer sparsity and basically require an LLM agent to assist interpretation. The author states that their paper found neuron activations can be sparsified to an L0 of about 32 per layer per token with a KL of about 0.5, whereas transcoders at the same L0 are about 2.0; and states that per-neuron steering on MoE models performs poorly.
Related discussion
- Older than 72 hours · Not included in the 72-hour figures
- RedditReddit r/singularityNeurons as an Interpretable Basis(opens the original post in a new tab)
- X@Yamamoto_NeuroXGood news about interpretability: Goodfire AI presents positive results with concrete experiments and data: "Artificial intelligence neurons are surprisingly clean, and when viewed on a computational-function basis, they are quite easy to handle." Qwen3.6 35B…(opens the original post in a new tab)
11 likes·Nov 11 12:45Collect
- X@NeoAIForecast@GoodfireAI doing the research!! They studied thousands of neurons in open-weight Qwen3.6 35B-A3B and says most have j…(opens the original post in a new tab)
- X@NFT_Chen🔥Wow! Dissecting neurons with Qwen3.6-35B: cutting 92% of weak activations, performance barely drops! A slight fine-tune fully restores it! The AI black box is being opened! Previously it seemed neurons in neural networks were a chaotic mess, with one neuron handling several unrelated things at once. Now the Goodfire team directly takes Qwen3.6-…(opens the original post in a new tab)
13 likes·Nov 11 06:14Collect
- X@yobai_duanGoodfire today released a technical report on neuron interpretability, supplementing an October opinion article, and opened 1,500 neuron descriptions for Qwen3.6-35B-A3B. The team's experiments show that directly studying neurons can be used to predict and simulate some strongly activated behaviors. The results come from a single model and have not yet been independently reviewed, worth…(opens the original post in a new tab)
2 likes·Nov 11 05:46Collect
- X@AGTPinsightsGoodfire AI just released the first technical report showing LLM neurons are interpretable and practical. Here's what y…(opens the original post in a new tab)
- X@AIQuantingGoodfire published a research article on November 10 called "Neurons as an Interpretable Basis". It is a companion to t…(opens the original post in a new tab)
2 likes·Nov 11 06:14Collect
- XX @AndrewCurran_The technical update to Neural Networks Are Already Clean has arrived!(opens the original post in a new tab)
- X@eric_hothe long awaited technical follow-up to Neural Networks are Already Clean is here! it dives deep into detailed neuron…(opens the original post in a new tab)
100 likes·Nov 11 06:14Collect
- XX @GoodfireAIIn Neural Networks are Already Clean, @nickcammarata argued that neurons were far more legible than previously thought…(opens the original post in a new tab)
775 likes·Nov 11 06:14Collect
Attention trend
New sources per day
- Nov 7: 1 new sources (1 outlets, 0 accounts)
- Nov 8: 0 new sources (0 outlets, 0 accounts)
- Nov 9: 0 new sources (0 outlets, 0 accounts)
- Nov 10: 1 new sources (1 outlets, 0 accounts)
- Nov 11: 9 new sources (0 outlets, 9 accounts)
- Nov 12: 0 new sources (0 outlets, 0 accounts)
- Nov 13: 1 new sources (0 outlets, 1 accounts)
- Nov 14: 0 new sources (0 outlets, 0 accounts)
- Nov 15: 0 new sources (0 outlets, 0 accounts)
- Nov 16: 0 new sources (0 outlets, 0 accounts)
- Nov 17: 0 new sources (0 outlets, 0 accounts)
- Nov 18: 0 new sources (0 outlets, 0 accounts)
- Nov 19: 0 new sources (0 outlets, 0 accounts)
- Nov 20: 0 new sources (0 outlets, 0 accounts)