Skip to content
EventPast event

Goodfire report claims neurons can serve as interpretable units

3 reports2 reporting sourcesLatest report

Start here

AI overview

LessWrong user neverix commented on Goodfire's "Neural Networks Are Already Clean," arguing that the neuron-based interpretability agenda is worth pursuing but expressing reservations about the results on LLMs and waiting for a formal paper; they said the neuron attribution graph is equivalent to an overlay composed of per-layer probes and steering vectors, which can provide a handle for observing and intervening in the model, but is insufficient to explain internal mechanisms in detail, and the attribution graph has poor sparsity, basically requiring LLM agents to assist interpretation. Goodfire Research subsequently released a technical report, "Neurons as an Interpretable Basis," as a follow-up to Nick Cammarata's widely shared essay, using sparsification, attribution tools, and research agents to analyze Qwen3.6-35B-A3B, arguing that neurons themselves can serve as interpretable computational units: ablating the weakest 92.2% of MoE neuron activations in each forward pass increased language model loss by only 2.9%; natural language descriptions allowed a judge model to predict with 95% accuracy whether a neuron would fire, simulation recovered 73% of the ablation damage (measured against the recoverable upper bound), and it could write activation text stronger than any text in a 10B-token corpus. The appendix analyzed the Colorado/Tornado neuron and found that about 98.8% of the causal effect came from Colorado and other related-location text; Tornado was not a hidden secondary function; the report suspected this association arose from Qwen3-32B copying the initialization of Qwen3-14B layers, and considered this overlap benign.

Generated by AI from reports · Updated 7 days ago

Attention and developments

Attention, hourly

Now 39, -1% in 24 hours·Peak 42 (Nov 13 04:00)

Nov 14Nov 15Nov 16Now 39

Storyline

Developments on the same subject and issue, from earliest to latest

3 events
  1. · 11 reports· Attention41
    Cammarata and Goodfire claim most neurons are readable; researchers disagree
  2. · 1 reports· Attention20
    Neel Nanda comments on new work in neuron interpretability
  3. · 3 reports· Attention39

    Goodfire report claims neurons can serve as interpretable unitsThis event

Latest developments

2 developments
  1. · 2 reports
    Goodfire research: neurons can serve as interpretable computational units
    Goodfire Research: Neurons as an interpretable foundation
  2. · 1 report
    Commentary on Goodfire's neuron interpretability paper
    LessWrong: Views on Goodfire's neuron interpretability article "Already Clean"

Progress timeline

Nov 13
Nov 12
  1. LessWrong
    Goodfire report says Qwen3.6-35B-A3B neurons can serve as interpretable units

    Goodfire released the technical report "Neurons as an Interpretable Basis," arguing that neurons themselves can serve as interpretable computational units. As a follow-up to Nick Cammarata's widely shared essay, the report uses sparsification, attribution tools and a research agent to analyze Qwen3.6-35B-A3B, claiming that ablating the weakest 92.2% of MoE neuron activations in each forward pass increases language model loss by only 2.9%; that natural-language descriptions allow a judge model to predict with 95% accuracy whether a neuron fires; that simulation recovers 73% of the ablation damage (measured against the recoverable upper bound); and that it can write activation text stronger than any text in a 10B-token corpus. The appendix analyzes Colorado/Tornado neurons and finds that about 98.8% of the causal effect comes from Colorado and other related-location text, and that Tornado is not a hidden secondary function; the report suspects this association stems from Qwen3-32B copying the initialization of Qwen3-14B layers, and considers this overlap benign.

Nov 11
Nov 10
  1. Goodfire ResearchSelected
    Neurons as an interpretable foundation

    Goodfire Research studies the neurons of Qwen3.6-35B-A3B and proposes that neurons themselves can serve as interpretable computational units. After the team ablated the weakest 92.2% of MoE neuron activations in each forward pass, language model loss increased by only 2.9%; 69% of neuron poles are dominated by a single pattern for at least 90% of the KL damage. After describing neurons in natural language, a judgment model can predict with 95% accuracy whether they activate on given text, and recovers 73% of the ablation damage in high-activation regions; 94% of neurons can be activated by synthetic text to levels exceeding the highest in a corpus of more than 10B tokens.

Nov 7
  1. LessWrong
    Views on Goodfire's neuron interpretability article "Already Clean"

    LessWrong user neverix posted a commentary on Goodfire's article "Neural Networks Are Already Clean," which argues that neuron-based interpretability may be easier than expected; the author considers this agenda worth pursuing but is reserved about the results on LLMs and is waiting for a formal paper. The author considers neuron attribution graphs equivalent to an overlay layer composed of per-layer probes and steering vectors, which can provide a handle for observing and intervening in the model, but is insufficient to explain the model's internal mechanisms in detail; neurons are the model's native substrate, and cross-model application is cheaper than dictionary learning methods, but their attribution graphs have poorer sparsity and basically require an LLM agent to assist interpretation. The author states that their paper found neuron activations can be sparsified to an L0 of about 32 per layer per token with a KL of about 0.5, whereas transcoders at the same L0 are about 2.0; and states that per-neuron steering on MoE models performs poorly.

Related discussion

Total 10 items

Attention trend

New sources per day

12 in 14 days·most 9 (Nov 11)·today 0

  • Nov 7: 1 new sources (1 outlets, 0 accounts)
  • Nov 8: 0 new sources (0 outlets, 0 accounts)
  • Nov 9: 0 new sources (0 outlets, 0 accounts)
  • Nov 10: 1 new sources (1 outlets, 0 accounts)
  • Nov 11: 9 new sources (0 outlets, 9 accounts)
  • Nov 12: 0 new sources (0 outlets, 0 accounts)
  • Nov 13: 1 new sources (0 outlets, 1 accounts)
  • Nov 14: 0 new sources (0 outlets, 0 accounts)
  • Nov 15: 0 new sources (0 outlets, 0 accounts)
  • Nov 16: 0 new sources (0 outlets, 0 accounts)
  • Nov 17: 0 new sources (0 outlets, 0 accounts)
  • Nov 18: 0 new sources (0 outlets, 0 accounts)
  • Nov 19: 0 new sources (0 outlets, 0 accounts)
  • Nov 20: 0 new sources (0 outlets, 0 accounts)

Hourly attention

Current attention 39·Peak in the comparable period 42 (Nov 13 04:00)·Change over the comparable past 24 hours -1%

0204060Nov 1304:00Nov 1410:00Nov 1517:00Nov 1623:00