Neel Nanda comments on new work in neuron interpretability
Start here
Neel Nanda comments on a new work in neuron interpretability: he believes neuron attribution graphs are useful for practical problems such as investigating suspicious model behavior, but they do not count as a major interpretability advance. He says they may be better than SAEs and transcoders, but expects that after pruning to key components there will still be significant errors and omissions, and they also do not address superimposed information in weak neuron activations of LLMs. He still believes superposition holds, and says residual stream SAEs are extremely useful.
Generated by AI from reports · Updated 2026-10-16
Storyline
- · 11 reports· Attention41Cammarata and Goodfire claim most neurons are readable; researchers disagree
- · 1 reports· Attention20
Neel Nanda comments on new work in neuron interpretabilityThis event
- · 3 reports· Attention39Goodfire report claims neurons can serve as interpretable units
Progress timeline
- XX @AISafetyMemes· DiscussionIs today's interpretability "breakthrough" a big deal?(opens the original post in a new tab)
- X @NeelNanda5Neel Nanda comments on new neuron attribution graph work: considerable practical value, but not a major interpretability advance
Neel Nanda comments on a new neuron interpretability work: neuron attribution graphs are useful for practical problems like investigating suspicious model behavior, but do not count as a major interpretability advance. He thinks it may be better than SAE/transcoder, but expects that after pruning to key components there will still be significant error and omission, and it also does not handle superposition information in weak neuron activations of LLMs. He still believes superposition holds, and says residual stream SAEs are extremely useful.
Related discussion
- Older than 72 hours · Not included in the 72-hour figures
- XX @AISafetyMemesIs today's interpretability "breakthrough" a big deal?(opens the original post in a new tab)