Skip to content
LessWrong· Julian Bradshaw·Indexed by this site · Originally published· Alignment →

Goodfire report says Qwen3.6-35B-A3B neurons can serve as interpretable units

Linkpost: "Neural Networks Are Already Clean" Followup Technical Paper

AI summary

Goodfire released the technical report "Neurons as an Interpretable Basis," arguing that neurons themselves can serve as interpretable computational units. As a follow-up to Nick Cammarata's widely shared essay, the report uses sparsification, attribution tools and a research agent to analyze Qwen3.6-35B-A3B, claiming that ablating the weakest 92.2% of MoE neuron activations in each forward pass increases language model loss by only 2.9%; that natural-language descriptions allow a judge model to predict with 95% accuracy whether a neuron fires; that simulation recovers 73% of the ablation damage (measured against the recoverable upper bound); and that it can write activation text stronger than any text in a 10B-token corpus. The appendix analyzes Colorado/Tornado neurons and finds that about 98.8% of the causal effect comes from Colorado and other related-location text, and that Tornado is not a hidden secondary function; the report suspects this association stems from Qwen3-32B copying the initialization of Qwen3-14B layers, and considers this overlap benign.

Read originallesswrong.com