Goodfire discovers neuron communities in language models
Discovering Neuron Communities in Language Models
Goodfire Research reports an unsupervised method that automatically discovers, on Llama-3.1-8B using 10 million prompts, small neuron communities that jointly implement model behavior; such communities often comprise fewer than ten neurons. The method first uses gradient-based attribution to find the neurons driving each prediction, connects neurons that repeatedly jointly drive the same prediction into a graph, then groups them using community detection, and tests necessity and sufficiency on held-out prompts. In one case, six neurons across six layers support vowel-initial predictions on prompts ending in an; removing them shifts the prediction toward consonant-initial words, while keeping them and removing the remaining MLP neurons in those six layers largely preserves the original prediction. Other communities cover the grammatical choice between is/are after there, retrieval and computation, topic, format, common phrases, and token suppression; the method also reproduces on Qwen 3.6 35B-A3B. The authors note that sufficiency is limited to the layers containing the community's neurons, and how the community specifically computes and how it interacts with the rest of the model remain unexplained.
Goodfire can discover communities often comprising fewer than ten neurons in Llama-3.1-8B without prespecified functions, offering a scalable entry point for studying how computation is organized inside models.