Large-scale neuron community discovery reveals sparse neuron behavior in LLMs
Large-scale Neuron Community Discovery Reveals Neuron-Sparse Behaviors in Language Models
The researchers propose an unsupervised neuron community discovery method that finds groups of neurons reused across prompts and causally necessary and sufficient, within the layers they span, for common computations. The method combines gradient attribution, a graph connecting neurons that are repeatedly co-attributed, and community detection; on Llama-3.1-8B it discovers thousands of candidate communities using 10 million prompts, and evaluates them on held-out prompts with three causal tests: community necessity, sufficiency within a layer subset, and layer necessity. The discovered communities support functions such as syntactic agreement, token probability calibration, fixed phrases, in-context copying, addition and multiplication. When communities are capped at 128 neurons across all layers, 10.3% of held-out prompts have the contribution of at least one layer explained by neuron sparsity; the coverage is 8.7% when at least four layers are required, and 3.3% when at least twelve layers are required. The results show that small groups of existing neurons can serve as computational components reused across prompts and can be discovered at scale.