Views on Goodfire's neuron interpretability article "Already Clean"
Takes on “Already Clean”
LessWrong user neverix posted a commentary on Goodfire's article "Neural Networks Are Already Clean," which argues that neuron-based interpretability may be easier than expected; the author considers this agenda worth pursuing but is reserved about the results on LLMs and is waiting for a formal paper. The author considers neuron attribution graphs equivalent to an overlay layer composed of per-layer probes and steering vectors, which can provide a handle for observing and intervening in the model, but is insufficient to explain the model's internal mechanisms in detail; neurons are the model's native substrate, and cross-model application is cheaper than dictionary learning methods, but their attribution graphs have poorer sparsity and basically require an LLM agent to assist interpretation. The author states that their paper found neuron activations can be sparsified to an L0 of about 32 per layer per token with a KL of about 0.5, whereas transcoders at the same L0 are about 2.0; and states that per-neuron steering on MoE models performs poorly.