Skip to content

Large-scale neuron community discovery reveals sparse neuron behavior in LLMs

Large-scale Neuron Community Discovery Reveals Neuron-Sparse Behaviors in Language Models

arXiv· arXiv:2611.23112· Igor Shilov, Thomas Fel, Tal Haklay, Lucius Bushnaq, Owen Lewis·Indexed by this site · 2 days ago· Originally publishedAlignment· Its place in the scan record →
AI summary

The researchers propose an unsupervised neuron community discovery method that finds groups of neurons reused across prompts and causally necessary and sufficient, within the layers they span, for common computations. The method combines gradient attribution, a graph connecting neurons that are repeatedly co-attributed, and community detection; on Llama-3.1-8B it discovers thousands of candidate communities using 10 million prompts, and evaluates them on held-out prompts with three causal tests: community necessity, sufficiency within a layer subset, and layer necessity. The discovered communities support functions such as syntactic agreement, token probability calibration, fixed phrases, in-context copying, addition and multiplication. When communities are capped at 128 neurons across all layers, 10.3% of held-out prompts have the contribution of at least one layer explained by neuron sparsity; the coverage is 8.7% when at least four layers are required, and 3.3% when at least twelve layers are required. The results show that small groups of existing neurons can serve as computational components reused across prompts and can be discovered at scale.

Read originalarxiv.org