Skip to content
EventPast event

Neel Nanda comments on new work in neuron interpretability

1 reports1 reporting sourcesLatest report

Start here

AI overview

Neel Nanda comments on a new work in neuron interpretability: he believes neuron attribution graphs are useful for practical problems such as investigating suspicious model behavior, but they do not count as a major interpretability advance. He says they may be better than SAEs and transcoders, but expects that after pruning to key components there will still be significant errors and omissions, and they also do not address superimposed information in weak neuron activations of LLMs. He still believes superposition holds, and says residual stream SAEs are extremely useful.

Generated by AI from reports · Updated 2026-10-16

Storyline

Developments on the same subject and issue, from earliest to latest

3 events
  1. · 11 reports· Attention41
    Cammarata and Goodfire claim most neurons are readable; researchers disagree
  2. · 1 reports· Attention20

    Neel Nanda comments on new work in neuron interpretabilityThis event

  3. · 3 reports· Attention39
    Goodfire report claims neurons can serve as interpretable units

Progress timeline

Oct 16
  1. X @NeelNanda5
    Neel Nanda comments on new neuron attribution graph work: considerable practical value, but not a major interpretability advance

    Neel Nanda comments on a new neuron interpretability work: neuron attribution graphs are useful for practical problems like investigating suspicious model behavior, but do not count as a major interpretability advance. He thinks it may be better than SAE/transcoder, but expects that after pruning to key components there will still be significant error and omission, and it also does not handle superposition information in weak neuron activations of LLMs. He still believes superposition holds, and says residual stream SAEs are extremely useful.

Related discussion

Total 1 items