A toy model of superposition
Toy Models of Superposition
Quick read
Based on the paper's full text (AI-generated); refer to the original
A paper released by Anthropic researchers used small ReLU networks on synthetic sparse feature data to demonstrate superposition, i.e., the model can represent more features in nearly orthogonal directions in fewer dimensions than the number of features. Experiments show that the sparser the features, the more the model tends to encode additional features in non-orthogonal directions and tolerate the resulting interference, and whether a feature enters superposition is controlled by a phase transition. Under uniform superposition, features organize into geometric structures such as antiparallel pairs, triangles, pentagons, and tetrahedra; the authors connect this to uniform polyhedra and the Thomson problem, and discuss possible connections to adversarial examples and grokking as well as implications for mechanistic interpretability.
- Neural networks often mix multiple unrelated concepts in the same neuron (polysemanticity), making interpretability research difficult. The authors hope to understand why and when models use fewer dimensions than the number of features to represent more features.
- The authors construct small ReLU networks, train them on sparse synthetic feature data, observe whether the model stores more features than dimensions in non-orthogonal directions (i.e., superposition), and analyze its phase transitions, geometric structure and learning dynamics.
- Experiments show that superposition does occur, both monosemantic and polysemantic neurons can form, and some computations (such as absolute value) can be performed under superposition; whether a feature enters superposition is controlled by phase transitions and organized into geometric structures such as digons, triangles, pentagons and tetrahedra; preliminary connections with adversarial examples, grokking and MoE are also found.
- The authors emphasize that these conclusions come from a toy model, and the extent to which they can be generalized to real networks is very uncertain, especially the geometric and learning dynamics results; how to detect, control or eliminate superposition, and how feature importance and sparsity curves can be estimated in real models, remain open questions.
The paper uses a reproducible toy model to demonstrate superposition and its phase transitions and geometric structure, providing a transferable analytical framework for understanding the polysemanticity of neurons in large models.