Study reveals weakening-type neurons with outsized influence in Transformers

What is the relationship between the direction a GLU neuron reads from the residual stream and the direction it writes back, and how does this relationship correspond to activation frequency and model behavior.

• Scenario: The authors place the work in mechanistic interpretability, focusing on individual neurons in the MLP sublayer. Prior work either classifies by activation context (e.g., Voita et al., 2024) or defines functional roles by output weight direction (e.g., Gurnee et al., 2024). The authors argue that neither fully captures the mechanism implemented by a neuron: a neuron is determined not only by when it activates and what it writes, but also by the relationship between the direction it reads from the residual stream and the direction it writes back.
• Existing gaps: The authors state that Elhage et al. (2021) hypothesized that negative cosine of input–output weights might implement memory management, and Gurnee et al. (2024) computed such cosines for non-GLU GPT-2 neurons but did not interpret the results. The authors claim that prior work has not established an IO classification for gated activations (Shazeer, 2020), nor systematically analyzed the layer distribution of IO classes in recent LLMs and their impact on behavior. SwiGLU/GEGLU are used by Llama, OLMo, Qwen, Gemma, and others.
• Core observation: If the output weight is opposite in direction to the input weight, the neuron weakens the direction it detects in the residual stream; the authors call this a weakening neuron. They further distinguish strengthening, conditional strengthening/weakening, proportional change, and orthogonal output.
• This paper's approach: Propose an IO analysis of gated neurons based on weight cosine, count the layer distribution and activation frequency of each class across multiple LLMs, and use ablations and case studies to examine the impact of weakening neurons on output distributions, including cases where the gating value is negative.

In the introduction and Related Work, the paper groups prior work into several categories and notes that the IO perspective is relatively rare.

Residual Stream and Linear Representations

• Elhage et al. (2021): proposed the residual stream; the authors note that their footnote mentioned negative IO cosine might do memory management, but did not pursue it. Belrose et al. (2025), nostalgebraist (2020) view residual states as intermediate guesses about the next token; Rushing and Nanda (2024) discuss this as the iterative inference hypothesis. Park et al. (2024) discuss several aspects of the linear representation hypothesis; Lad et al. (2024) define inference stages.

Neurons, Superposition, and SAE

• Single neurons and critiques: Miller and Neo (2023), Dai et al. (2022), Niu et al. (2024) attempt to understand individual neurons. Millidge and Black (2022) find interpretable directions not based on individual neurons. Elhage et al. (2022) argue that interpretable features are non-orthogonal, superposed directions in model space, corresponding to sparse linear combinations of neurons in MLPs. This leads to SAE (Sharkey et al., 2022). Recent work by Kantamneni et al. (2025), Mueller et al. (2025), Wu et al. (2025), Arora et al. (2026) points out that SAE does not always outperform baselines; Leask et al. (2025) argue its features are not "canonical." Gurnee et al. (2023) argue that interpretable features correspond to sparse combinations of neurons, including individual neurons. Zhao et al. (2026)'s culture-sensitive neurons and Ali et al. (2025)'s prominent but detrimental neurons are still searching for new neuron classes.
• Activation context and output direction: Voita et al. (2024) classify by activation context and find token detectors; Gurnee et al. (2024) define functional roles such as suppression neurons by output weights. Semantic-level IO: Geva et al. (2021) view neurons as key-value memory; Gur-Arieh et al. (2025) use feature outputs to help guess their activation sequences. McDougall et al. (2024), Elhelo and Geva (2025) do weight-based input-output token analysis of attention head OV circuits. The authors state that they compute the cosine of input and output weights at the mathematical level, and gating makes IO more complex.

Suppression, Erasure, and Confidence

• Memory management and erasure: The authors state that Lad et al. (2024) seem to conflate several mechanisms as residual sharpening. Heimersheim and Turner (2023) find that overwriting can also do memory management. Janiak et al. (2024) observe attention heads erasing the outputs of previous heads; McGrath et al. (2023) observe MLPs partially erasing what earlier units wrote; Rushing and Nanda (2024) find sparse erasure neurons and hypothesize they are related to Gurnee et al. (2024)'s suppression neurons.
• Confidence: McDougall et al. (2024)'s copy suppression heads suppress cross-position copying; the authors state this differs from memory management because the suppressed representation is not needed by earlier stages, and it is thought to improve calibration. Stolfo et al. (2024), building on Gurnee et al. (2024), find entropy neurons and token frequency neurons, classified by output; the authors state these are therefore a priori different from this paper's IO classes. Lv et al. (2024) find an anti-overconfidence mechanism in the last layer; Joshi et al. (2025) confirm a confidence correction stage in the last layer and find a calibration direction in early layers. Stacey et al. (2026) analyze the robustness of hidden-layer uncertainty probes. The authors state that none of this work connects confidence with IO cosine, and in Section 9 they hypothesize a connection.

Baseline methods are not external attack or defense methods. The randomness control is i.i.d. Gaussian vectors, as well as layer-mismatched cosines (Section 4.3, Section F). The ablation control is randomly sampling the same number of neurons from the same layer as the weakening neurons. Activation statistics use a random subset of Dolma.

The authors position this paper as: establishing an IO classification for gated neurons and systematically examining the distribution of these classes in recent LLMs and their impact on behavior; this differs from classifications that look only at activation context or only at output direction, and also differs from the memory management hypothesis that explains only negative cosine, because they simultaneously observe a large number of strongly positive cosines.

The overall approach is IO functionality classification: for each neuron, compare the direction it reads from the residual stream with the direction it writes back. The object of analysis is the neuron weights, not activations, and not transcoder features.

Problem setup and gated neurons

• Residual read-write: following Elhage et al. (2021), a unit reads from the residual stream and then writes back. The update of a SwiGLU neuron is Swish(⟨wgate, x⟩)·⟨win, x⟩·wout. wgate and win are the directions that cause activation, and wout is the write-back direction. GEGLU replaces Swish with GELU, and the rest is the same.
• Difference from ReLU: a traditional MLP is Wout ReLU(Win x). SwiGLU requires two sets of input weights: Wout (Swish(Wgate x) ⊙ (Win x)). The gate function can output any positive or negative value; for example, when xgate>0 and xin≪0, the SwiGLU output is far less than 0. The authors say Swish and GELU are often regarded as smooth approximations of ReLU, and negative gate values were previously often thought to mainly serve training dynamics.
• Semantics only as intuition: the authors say "detect a concept then write out a concept" is not a necessary assumption for the classification. The method chooses purely weight cosine, so as to describe the mathematical function implemented by the neuron in terms of its update to the residual stream.

IO classification of GLU

• Two non-GLU classes of input manipulation: cos(win, wout) very negative is weakening (detect a direction then write its opposite direction); very positive is strengthening.
• GLU prototype classes (cosine about ±1 or 0, Fig. 1, Section 4.2.2): (i) strengthening / weakening: the three weight vectors are roughly collinear, and cos(win, wout)≈±1. (ii) conditional strengthening / weakening: win and wout are roughly collinear, and wgate is orthogonal to them; whether it strengthens or weakens depends on whether wgate appears in the residual stream. (iii) proportional change: wout is collinear with wgate and orthogonal to win; if wgate appears, write a multiple of that direction, and the multiple is proportional to the occurrence of win. Opposed to these "input manipulations" is orthogonal output: writing to a direction not detected by the input.
• Three granularities for the general case: (1) assign to the nearest prototype, threshold τ=±0.5; (2) plot the marginal distributions of the three cosines; (3) make a scatter using the three weight cosines. If one only knows that cos(win, wout) and cos(wgate, wout) are both greater than 0.5, one cannot infer cos(win, wgate)>0.5; such mismatched class names are prefixed with atypical. The complete decision table is in Table 2 of Section E. Such neurons exist in Fig. 3, but are overall few.

Random baselines and implementation

• Two random baselines (Section 4.3): (i) i.i.d. Gaussian vectors, corresponding to random initialization; (ii) per-layer mismatched cosines. Details in Section F. Fig. 4 marks the 95% random region of each of the three cosines with red lines; variant (i) is dotted and (ii) is dashed, and the authors say the two nearly coincide.
• Implementation: the code is based on TransformerLens v2. When loading the model, a preprocessing that does not change behavior is done to make the weights more interpretable; the authors use a custom version with additional preprocessing (Section D). The authors say the weight preprocessing makes cos(wgate, win) of weakening neurons positive (Section D.2). The authors plan to release the code; the main text gives the repository name, and the address is not recorded here.

Activation statistics and ablation definitions

• Activation frequency: count the frequency of xgate>0, and compute its Pearson correlation with cos(win, wout). The sample size is the number of MLP neurons of that model. The corpus is a random 20M token subset of Dolma (following the scale of Voita et al., 2024).
• Ablation scale: the number of neurons ablated each time equals the number of weakening neurons of that model. OLMo-7B has 526 weakening neurons, so each round ablates 526. The control is randomly drawing the same number from the same layers. Two ablations: zero ablation (set activation to 0) and mean ablation (set to the mean activation of that neuron, Section G.3). Section 7 of the main text mainly uses mean ablation.
• Four quadrants of conditional ablation (the definition is simplified in the main text; the precise statement is in Section D.2): (i) xgate>0, xin>0, so xpost>0; (ii) xgate>0, xin<0，从而 xpost<0；(iii) xgate<0、xin<0，从而 xpost>0; (iv) xgate<0、xin>0, so xpost<0. Only activations satisfying the corresponding sign condition are ablated.

Methods used for case analysis

• Two existing types of methods: projecting weights into vocabulary space (Geva et al., 2022; Dar et al., 2023, etc.), and finding text that strongly activates the neuron (Geva et al., 2021; Nanda, 2022, etc.). The authors say the cases only show what the IO classes can do, and are not used to support statistical claims. On OLMo-7B, strengthening neuron 28.4737 and weakening neuron 31.9634 are chosen (zero-indexed layer.neuron; the model has 32 layers, and the latter is in the last layer).

Experimental Setup

• Models: 24 non-GLU transformers in total, covering encoder-decoder, encoder-only, and decoder-only, with ReLU and GELU activations, and training data including language and non-language; the list is in Section C.1, results in Section J.3.1. 12 GLU models in total, including SwiGLU and GEGLU, list in Section C.2. Fig. 2 plots 9 of these 2B–9B models: OLMo-7B-0424, Gemma-2-2B, Gemma-2-9B, Llama-2-7b, Llama-3.1-8B, Llama-3.2-3B, Mistral-7B-v0.1, Qwen2.5-7B, Yi-6B; those smaller than 2B are in Appendix Fig. 26.
• Corpus: Activation frequency and ablations use a random 20M tokens of Dolma. The authors note that Dolma is OLMo's training data; Gemma and Llama have no public training sets, so Dolma may be only a subdistribution (Dolma is English and code only, while Llama is trained on multilingual data). The authors consider this subdistribution sufficient for the purpose here. Frequency experiments use OLMo-7B, Gemma-2-2B, Llama-3.1-8B, Llama-3.2-3B. To save resources, ablations are done only on OLMo-7B and Llama-3.2-3B; the main text mainly shows OLMo, and states that Llama results are similar and in the supplementary materials.
• Metrics: Median cos(win, wout) within layers; class counts by τ=±0.5; gate>0 frequency and its Pearson correlation with cos(win, wout) (p<0.01）；消融前后的 loss 与输出分布熵。熵图的横轴为 entropy(clean)−entropy(ablated)，单位为 nats。
• 消融协议：每轮消融数等于 weakening 神经元数（OLMo-7B 为 526）。对照为同层随机神经元。正文聚焦 mean ablation；zero ablation 见 Section G 与 J.2。重复次数论文未报告。
• 随机区域：Fig. 4 的红线是三个余弦的 95% 随机区域。

主结果

Fig. 2 中，九个 2B–9B 模型的中位 cos(win, wout) 在网络前部为正、在末尾为负。Fig. 3（Llama-3.2-3B，精确计数见 Table 12）中，input manipulator 占全部神经元的 25%，在第 7–11 层（零起点）可达 50%。作者称 Llama 中超过 80% 的 input manipulator 属于 conditional strengthening 这一类。Table 1 给出 cos(win, wout) 与 gate>0 frequency correlation:

Model	Pearson correlation	Source
OLMo-7B-0424	-0.91	Table 1
Gemma-2-2B	-0.76	Table 1
Llama-3.1-8B	-0.59	Table 1
Llama-3.2-3B	-0.54	Table 1

• Layer distribution: The authors state that early–middle layers are dominated by conditional strengthening, while late-stage weakening occupies a large share of the fewer input manipulators in late layers; proportional change is another important class of input manipulators in late layers. In Fig. 4, layers 0 and 14 have many points above the x-axis, while the cluster at layer 27 moves below; lighter points in the lower-left are weakening. The authors state that even neurons labeled as orthogonal output often have cosine beyond the significance threshold, so they still do input manipulation to some degree; for example, many neurons in layer 14 have cos(win, wout) below 0.5 but above the significance threshold.
• Activation frequency: The authors state that, consistent with Gurnee et al. (2024), (conditional) strengthening activates rarely, while (conditional) weakening activates frequently. OLMo's correlation of −0.91 is stronger than the highest of about −0.69 they cite from Gurnee et al. (2024). By layer (Fig. 10), the effect mainly comes from middle layers. The authors state this is preliminary evidence that weakening, though a small class, is disproportionately important. Fig. 5 is a heatmap for OLMo-7B; the authors state that darker colors indicate more neurons in that region, and annotate corr: -0.91, p<0.01.
• Exact bar lengths in Fig. 3: The paper says see Table 12. The main-text transcription has all class names, but exact counts per layer are not aligned in a verifiable table body, so specific bar lengths are not filled here.

Ablations: Entropy and Negative Gating

• What was measured: On 20M tokens, for OLMo-7B (and Llama-3.2-3B in the supplementary materials), mean ablation by IO class, comparing output entropy and loss. Weakening is compared against same-layer random baselines and other IO classes.
• Results: The authors state that weakening has the highest effect on the measured metrics, despite only a few hundred neurons. Fig. 6 upper-left: over about 10^3 next-token predictions, weakening neurons reduce entropy by about 10 nats, with fewer cases of entropy increase. The authors state that strengthening often sharpens the distribution (reduces entropy in Fig. 13); weakening both often flattens the distribution and more often sharpens it. Other classes do not have such large effects (Fig. 17).
• Authors' interpretation: The authors expected that removing information from the residual stream would flatten the output, but weakening more often sharpens the distribution. Conditional ablation shows that sharpening, and flattening in another context, largely come from quadrants (ii) and (iii) (Fig. 6 middle-right, lower-left). Quadrant (iii) (xgate<0 and xin<0) is especially surprising to the authors: negative gating is relatively rare in weakening (Section 6), and due to Swish, negative gating values are relatively small. The authors state this is the first time (simultaneously with Kong et al., 2026, who focus on a different phenomenon) that negative gating values have a strong effect on model behavior, so for mechanistic interpretability Swish cannot be reduced to ReLU. The region of quadrant (ii) is small in weight space, but the authors state the residual stream still enters that region often enough. Quadrant (iv) has small and rare activations, and ablation has little effect (Fig. 6).

Case Studies: Entropy Reduction and Two Neurons

• Extreme entropy reduction sample (most extreme in quadrant (iii) under zero ablation): input ends with “the O”, correct next token is “mic”. Clean run predicts correctly. Relative to the ablated run, the clean run boosts the score of mic and similar tokens by up to 12 points, with no token's score dropping by a comparable amount. The authors state that at least in this example, quadrant (iii) sharpens the distribution by raising the correct next token.
• Indirect and cumulative: The authors state that no single weakening neuron alone achieves the effect, but a small subset of 8 weakening neurons can. Direct effects do not match the observed effect, so the relevant effect is indirect. A fuller mechanism is left for future work.
• Two OLMo neurons: 28.4737 (strengthening) further promotes review when the residual stream already indicates review should be the next token. 31.9634 (weakening) has weights corresponding to: producing “again” when the residual stream contains “minus again”; but strongly activating samples have no obvious semantic relation to again. It weakly positively activates when again is already a plausible continuation (e.g., after once), when xgate<0 and xin<0. The authors state that before the last MLP layer, again is already weakly present in the residual stream, and this neuron strengthens it. The authors argue that even when output weights are interpretable, the overall behavior of strengthening and weakening differs, with weakening being more complex; and they believe this relates to the nature of weakening: at least in quadrant (i), there is an apparent conflict between the intermediate prediction and what the neuron promotes.

Other Ablations and Analyses

• Layer-wise correlation (Fig. 10): the negative correlation mainly comes from middle layers; the claim about all but the last two layers is in the Section 9 summary, where the authors state that except for the last two layers, IO function and activation frequency are strongly correlated.
• Zero ablation entropy and loss results are in Sections G, J.2; the main text does not give another set of exact numbers aligned with Fig. 6, so none are cited here.
• Results for the 24 non-GLU models are in Section J.3.1; the authors state that within each group, results are consistent across models, and there is always a considerable number of weakening neurons, with many other neurons' IO cosine also far from 0. Specific medians are not given in the main-text figures.
• Fig. 4 and Appendix Figs. 30, 31 (Yi-6B, layers 24–31) repeat the pattern of “middle-layer cluster above the x-axis, last few layers move below”. The coordinate values in the appendix scatter plots cannot be read point-by-point from the transcription.
• The two 95% bounds of the random baseline nearly coincide in Fig. 4.
• Section H: The authors find examples of near-orthogonal vectors that represent very similar concepts (double checking), to illustrate that weight cosine should not be taken one-to-one as semantic similarity.

Limitations and Future Directions Noted by the Authors

• Mechanism remains a hypothesis: Section 9 states that the explanations for the findings are speculative hypotheses, not conclusive explanations, and the authors will attempt to confirm them in future work (Section 9).
• Entropy reduction mechanism: Section 7.3 states that in this example, the entropy reduction from weakening comes from indirect and cumulative effects; a fuller understanding of the mechanism is left for future work (Section 7.3).
• Scope of weight cosine: Limitations states that the method focuses on single-neuron parameter interpretation, is simple and computationally cheap, but its scope is inherently limited; it is designed as a complement, not a replacement, for other neuron analysis methods (Limitations).
• Cosine is not semantics: Limitations states that mathematical similarity of weights is suggestive but should not be taken as a one-to-one correspondence with semantic similarity; Section H has examples of near-orthogonal yet conceptually similar vectors (Limitations).
• Dependence on residual connections: Limitations states that the IO method assumes representations across layers are comparable to some degree, which is guaranteed by residual connections; if training without residual connections becomes standard (Ji et al., 2026), the method would no longer apply; it also cannot be directly applied to models like DeepSeek-V4 that use hyper-connections (Limitations).
• Ablations do not show intermediate steps, and understanding of later layers is limited: Limitations states that ablations show effects at the next-token prediction level for different IO classes, but do not show the intermediate steps from neuron activation to model output. A specific limitation of conditional ablation is that it does not apply when applied to several (transcription interrupted here) later layers, and therefore does not apply to most weakening neurons; the authors thus say understanding of weakening neurons remains limited (Limitations). The same cosine analysis on Transcoder is explicitly deferred to future work (Section 4.1).

Experimental Coverage

• Weight cosine covers 24 non-GLU models (Section C.1, J.3.1) and 12 GLU models (Section C.2); Fig. 2 shows the median cos(win, wout) for 9 of these 2B–9B models.
• gate>0 frequency and Pearson correlation are reported for OLMo-7B, Gemma-2-2B, Llama-3.1-8B, Llama-3.2-3B, on a random 20M tokens of Dolma (Section 6, Table 1).
• The main-text ablations are primarily mean ablations on OLMo-7B, with 526 neurons per round, including same-layer random controls and four activation quadrants; results for Llama-3.2-3B are stated by the authors to be in the supplementary materials (Section 7).
• Case studies analyze OLMo-7B's 28.4737 and 31.9634 in detail, and state that Section I has more neurons; the authors state that case studies do not support statistical claims (Section 8).
• The paper reports Pearson correlation p<0.01 and a 95% random region in Fig. 4; it does not report the number of ablation repetitions. The Dolma used on Gemma and Llama is written by the authors as possibly only a subdistribution of their training distributions (Section 6).

This analysis work uses the cosine of the input weights and output weights to classify the read-write function of GLU neurons, and examines the layer distribution, activation frequency, and effect on the next token distribution of these classes.

• Problem: the direction in which a neuron writes back to the residual stream can be in the same direction as, opposite to, or orthogonal to the direction it reads. Gated models also have a third vector, wgate. The authors hold that looking only at the activation context or only at the output direction does not systematize this read-write relationship.
• Classification: when the three weights are roughly collinear and cos(win, wout) is close to +1 or −1, they are strengthening and weakening respectively; when win and wout are collinear while wgate is orthogonal, it is the conditional version; when wout is aligned with wgate and orthogonal to win, it is proportional change; the rest are near-orthogonal output. The practical classification uses a threshold of ±0.5, and mismatches are labeled atypical. The random controls are Gaussian vectors and per-layer mismatched cosines.
• Distribution: on the nine 2B–9B GLU models of Fig. 2, the median cos(win, wout) is positive in the early part and negative at the end. In Llama-3.2-3B, input manipulators account for 25%, reaching 50% in layers 7–11, and in Llama more than 80% of input manipulators are conditional strengthening. In late layers there is more weakening, along with proportional change.
• Frequency and ablation: on four models, cos(win, wout) is negatively correlated with the frequency of gate>0 (Table 1): OLMo-7B is −0.91, Gemma-2-2B is −0.76, Llama-3.1-8B is −0.59, Llama-3.2-3B is −0.54, all p<0.01. OLMo-7B has 526 weakening neurons; mean-ablating them has a greater effect on output entropy than random neurons in the same layers and other IO classes. In Fig. 6, across about 10^3 predictions, entropy drops by about 10 nats. Quadrants (ii) and (iii) contribute greatly, and on this basis the authors hold that negative gate values affect behavior, and that Swish cannot be reduced to ReLU. In one extreme sample, the clean run raises the score of the correct token by at most 12 points, and the effect comes from the indirect accumulation of 8 neurons.
• The authors' conclusion: IO functions are broadly similar across the models examined, and can predict activation frequency; the small weakening class has a large effect on behavior, including the negative gate values previously thought not to affect transformer function. Section 9 conjectures that mid-layer conditional strengthening is confidence management for abstract concepts, and that the different quadrants of late-layer weakening are error correction, conflict resolution, or strengthening-like behavior, and states explicitly that these are only hypotheses. The authors hope IO analysis becomes part of the interpretability toolkit, while noting that it supplements rather than replaces other methods, and that it relies on the residual connection.