Skip to content

Anthropic Fellows paper: most transcoder features are actually MLP neurons, and neurons may be sufficient for circuit analysis

Paper Tracker· arXiv:2610.24521· Stepan Shabalin, Mateusz PiotrowskiAnthropic Fellows; Georgia Institute of Technology; AnthropicIndexed by this site · 2026-10-21· Originally publishedAlignment

Quick read

Interpretability

Based on the paper PDF (AI-generated); refer to the original

The authors compare sparsified MLPs on 13 open-source transcoders, and say that late-layer activations mostly come from features aligned with individual neurons.

Questions
MLP neurons are widely regarded as polysemantic, so circuit analysis has turned to transcoders. Arora et al. and Ye et al. show that thresholded neurons can also recover sparse circuits. The authors ask what transcoders contribute beyond out-of-the-box neurons.
Methods
Split the gated MLP by sign into encoder/decoder and do top-k. neuron feature refers to an encoder and decoder that simultaneously align to the same neuron, with a cosine threshold of 0.6 or 0.7. Then compare Jacobian, cross-seed reproduction, and movement relative to initialization.
Experiments and results
After linear correction, sparse neurons have lower KL than the transcoder, and higher FVU. Gemma 3 27B at 66% depth, NF accounts for 82.8% of activations (Table 4). On 1500 bigrams Gemmascope hits 75.6%, neurons 0.0%.
Limitations and future work
The authors state the limitations: base models and mainly English data, and per-layer transcoders on 1B–27B dense models, and only high-magnitude single-token activations are analyzed. No systematic human evaluation was done. The authors call the cross-layer and MoE results preliminary.
Experimental setupGemma 2 2B、Gemma 3 1B et al. · Kantamneni et al. 113 binary tasks、1500 FineWeb phrase bigrams et al. · FVU、KL divergence et al.
Evaluated models
Gemma 2 2BGemma 3 1BGemma 3 4BGemma 3 12BGemma 3 27BGemma 3 27B-itQwen 3 0.6BQwen 3 1.7BQwen 3 4BQwen 3 8BQwen 3 14BLlama 3.2 1BLlama 3.1 8BGemma 4 26B-A4BQwen 3.5 35B-A3B
Benchmarks
Kantamneni et al. 113 binary tasks1500 FineWeb phrase bigramsFineWebSmolLM2Pile-10kPride and PrejudiceTruthfulQAMMLU
Metrics
FVUKL divergenceReplacement ScoreAUCsort eval accuracyJacobian alignmentneuron feature fraction
AI summaryA paper by Anthropic Fellows found that a considerable portion of transcoder features strongly align with a single MLP neuron, which the authors define as neuron features, and states that they account for a considerable proportion of active features in the later layers of 13 open-source transcoders. The paper points out that only neuron features have a stable correspondence with the underlying MLP: other features vary more between different training runs, do not match the MLP's Jacobian, and change less relative to random initialization. Based on this, the authors argue that MLP neurons are stronger as interpretability primitives than is usually assumed, and call for improved methods for attributing the non-neuron structures used by models. The paper is 60 pages in total and was published at the NeurIPS InterpScience workshop.

A paper by Anthropic Fellows found that a considerable portion of transcoder features strongly align with a single MLP neuron, which the authors define as neuron features, and states that they account for a considerable proportion of active features in the later layers of 13 open-source transcoders. The paper points out that only neuron features have a stable correspondence with the underlying MLP: other features vary more between different training runs, do not match the MLP's Jacobian, and change less relative to random initialization. Based on this, the authors argue that MLP neurons are stronger as interpretability primitives than is usually assumed, and call for improved methods for attributing the non-neuron structures used by models. The paper is 60 pages in total and was published at the NeurIPS InterpScience workshop.

Related events and news

1 events · 1 news mentions

News that mentions this paper

Goodfire 报告称神经元可作可解释单元

  • Goodfire Researchcites the paper's id or link
    神经元作为可解释的基础
    Comparing Sparsified MLPs and Transcoders: Many Transcoder Features Are Neurons in Disguise.

Paper analysis

6 questions; open one to read the full answer

  1. What problem does this paper attempt to solve?

    The authors ask: must circuit analysis use transcoders, or are sparsified MLP neurons already sufficient.

    What does the transcoder contribute more to circuit analysis, relative to MLP neurons that are sign-split and top-k sparsified?

    • Setting: The authors state that polysemanticity is a standard obstacle to directly analyzing the MLP, and the field therefore replaces neurons with learned sparse dictionaries, and uses them for attribution graphs and frontier model behavior cases.
    • Tension in existing results: The authors state that Arora et al. (2026) under several settings can obtain circuits that are equally sparse and faithful using raw MLP neurons as with learned features; Ye et al. (2025) treat thresholded neurons as an SAE and can match learned dictionaries on standard benchmarks. Whether the transcoder is necessary therefore becomes a question.
    • The authors' question: If after sign splitting plus top-k neurons can already do circuits, is the extra dictionary term the transcoder learns computed in the corresponding model, or does it mainly improve reconstruction.
    • Core definition: The authors call transcoder features whose encoder, decoder, and activations all track a single MLP neuron neuron features, and state that after subtracting them, little structure aligned with the underlying MLP remains.
    • What this paper does: On 13 open-source transcoders, quantify the proportion of such features, Jacobian alignment, cross-seed reproduction, and movement relative to initialization, and compare sparsified neurons with the transcoder on replacement and probing.
  2. What related research exists?

    The authors divide related work into neuron circuits, transcoder attribution graphs, and known artifacts of dictionary training.

    Grouped into four groups per Section 4.

    Neuron circuits

    • Elhage et al. (2022) discuss polysemanticity with superposition toy models; Voita et al. (2024) and Gurnee et al. (2024) characterize polysemantic neurons in language models. Hanna et al. (2023) and Nikankin et al. (2025) document arithmetic and counting circuits via neurons.
    • Bills et al. (2023) through Choi et al. (2024) extend single-neuron automated description. The paper relays Arora et al. (2026): under several settings, appropriate attribution can make circuits sparse on a neuron basis. The paper relays Ye et al. (2025): treating thresholded neurons as an SAE can match learned dictionaries on standard benchmarks.

    SAE to attribution graphs

    • Bricken et al. (2023) and Cunningham et al. (2024) train SAEs on activations, Templeton et al. (2024) scale to frontier models. Dunefsky et al. (2024) propose the transcoder, approximating the MLP's input and output with a wider sparse replacement.
    • Marks et al. (2025), Ameisen et al. (2025), Lindsey et al. (2025b) build attribution graphs on such dictionaries. The paper writes that Gurnee et al. (2025) report feature families reproduced across model sizes, but do not compare with single neurons.

    Evaluation against learned features

    • Paulo et al. (2025a) report that learned features score higher on interpretability than single neurons; Paulo et al. (2025b) report that transcoders score higher than SAEs. The authors state that this paper is explaining this gap, not eliminating it.
    • Kantamneni et al. (2025) report that SAEs do not outperform a logistic regression baseline on 113 sparse probing tasks. Karvonen et al. (2025) give SAEBench. The paper states that comparisons under the attribution graph setting are found in Arora et al. (2026).

    Artifacts and this paper's baselines

    • Gurnee et al. (2023) show that concepts can often be read out from a small number of neurons. Dreyer et al. (2024), Kopf et al. (2025), Liu et al. (2026) cluster polysemantic neurons post hoc; the authors state that NFT and SNRL are training-time interventions. Regarding the splitting and absorption of Chanin et al. (2025), the authors state that it does not appear by default in their transcoder, but can be induced by the intervention in Section R.
    • Paulo and Belrose (2026) report that SAEs with different seeds are inconsistent on most features. The authors apply this analysis to transcoders, and state that the instability is concentrated in the non-neuron-aligned subset. Regarding the cross-layer unfaithfulness of Lange et al. (2026), the authors state that this paper's supporting evidence is limited.
    • The baselines are top-k neurons after sign splitting, plus recon, skip, and rank-1 correction. Comparison dictionaries include Gemmascope, the authors' TopK and JumpReLU, mwhanna, LXTC, and mntss. Tasks include 113 binary classifications, 1500 bigrams, FVU and KL, sort eval, and attribution graphs on Pile prompts.

    The authors place this paper after these works: what is MLP-aligned and more often reproduced across seeds is mainly neuron features; this is used to explain why automated interpretability scores are higher, rather than to claim the gap has disappeared.

  3. How does the paper address this problem?

    neuron feature refers to an encoder and decoder that simultaneously align to the same sign-split MLP neuron.

    The authors put sign-split and top-k sparsified gated MLPs and transcoders into the same encoder/decoder comparison, and call dictionary entries aligned with a single neuron neuron features.

    Directions of gated neurons

    • The object is a gated MLP. The hidden activation of the i-th neuron is written as ai=σ(gi)ui: g_i and u_i are the products of the i-th row of gate and up with the input, respectively.
    • At high activation it is approximately bilinear. When u_i>0 the effective encoder is Wup plus Wgate; when u_i<0 it is the difference of the two; the decoder is the signed Wdown. The two effective encoders are approximately orthogonal. Thereafter each neuron is viewed as two features by sign.
    • To align with transcoders, per-layer top-k is applied to the sign-split activations. The k used in the main text includes 32, 64, 128, and it is said that k≈50 can turn continuous activations into on/off; Section 5 suggests using about 64 for comparison.
    • For models without post-feedforward LayerNorm, activation magnitude rises with layer, so thresholds are taken per layer rather than using one absolute threshold for the whole model.

    Operational definitions

    • Quantitative criterion: the feature's encoder and decoder both exceed a threshold in cosine similarity with the same neuron's effective encoder and decoder. Figure 1 uses greater than 0.7. The main text says the threshold separating clusters from the main body is 0.6 to 0.9 across different transcoders, and later argues for using 0.6 or 0.7.
    • The authors also describe this correspondence using activation patterns, highest and lowest logits, and circuit roles; the proportion statistics mainly rely on weight cosine.
    • The early-layer proportion depends on initialization and training dynamics. The authors give one possible reason: early MLP neurons are more polysemantic, and the objective is less likely to select them as features.

    How faithfulness and stability are measured

    • Reconstruction uses FVU, i.e., normalized L2, the unexplained variance proportion. Replacement uses the KL of the replacement model relative to the original model, and top-1 accuracy. Attribution graphs use Replacement Score.
    • Each feature's contribution to the Jacobian is the rank-1 outer product of the decoder and the effective encoder, multiplied by the activation derivative at that token; summing over active features gives the transcoder's Jacobian. Alignment is the Frobenius cosine similarity with the MLP Jacobian.
    • Across seeds: only the initialization seed is changed. If in another run there exists a corresponding feature whose encoder and decoder cosines both exceed 0.7, it counts as reproduced.
    • Relative to initialization: the random weights at step 0 are saved, and the cosine of the trained encoder with the initial encoder is compared. TopK and JumpReLU have different encoder initialization scales; Appendix A writes out this difference.

    Training and two interventions

    • The authors' TopK uses Dion, with a learning rate of 10 to the power of negative 2; NFT and SNRL use 5 times 10 to the power of negative 3. k is annealed from 256 to 32, with annealing from step 1500 to 10000, for 15000 steps total. Batch is 256, sequence length is 128, and the data is SmolLM2.
    • JumpReLU uses another codebase, with a learning rate of 2 times 10 to the power of negative 4, sparsity coefficient 10, and 4 times 10 to the power of 8 training tokens. The sparsity term is a tanh loss with decoder norm weights.
    • Neuron initialization (NI): encoder rows are set to the sum or difference of gate and up with random signs, plus 10% Gaussian noise, the decoder is set to zero, and the rest of training is unchanged.
    • NFT gives each neuron G=16 sub-features; sub-features can enter the final selection only when the parent neuron is in its top-K, then r=4 is taken. SNRL pulls each feature's encoder and decoder toward a sparse reconstruction of at most 4 neuron directions, with coefficient λ=0.05.
    • The neuron replacement model trains a linear mapping on the pruned output, with optional skip, and a sign-gated rank-1 term. The loss is per-layer FVU plus KL, trained for 10000 steps on FineWeb. Optimizer details are in Appendix A.

    Evaluation protocol

    • Sparse probing follows Kantamneni et al. (2025): features are selected by the average activation difference across training groups, then L1 logistic regression is done, with k taken as 1, 2, 4, 8, 16, 32. The bigram probe constructs positive examples and missing-word negative examples on FineWeb, and checks whether the max-activation examples of the selected features contain the target phrase.
    • Sort eval follows Lindsey et al. (2024). Claude Haiku 4.5 looks at contexts of two groups of 10 passages each, at most 40 tokens, and judges which group the held-out sample belongs to. 500 times per configuration. The cache comes from 10M tokens of SmolLM2.
    • Attribution graphs treat neuron activations as the feature basis, using RelP and circuit-tracer. Appendix J uses encoder and decoder cosines both greater than 0.6 to label neuron features in the graphs.
  4. What experiments did the paper conduct?

    The authors state that in late layers neuron features contribute a large amount of activation, but the proportion varies with model and transcoder.

    Experimental setup

    • Models: Gemma 2 2B, Gemma 3 1B, 4B, 12B, 27B and 27B-it, Qwen3 0.6B, Qwen 3 1.7B, 4B, 8B, 14B, Llama 3.2 1B, Llama 3.1 8B, Gemma 4 26B-A4B, Qwen 3.5 35B-A3B. Appendix N additionally uses Qwen3-Coder-Next-Base. All are gated MLPs.
    • Dictionaries and baselines: the 13 open-source transcoders of Table 4, including Gemmascope, mwhanna, LXTC, and mntss cross-layer; the authors additionally train TopK and JumpReLU. The neuron baseline is per-layer top-k after sign splitting, optionally with linear reconstruction, skip, and rank-1.
    • Metrics: FVU, KL, Replacement Score, sparse probing AUC, sort eval accuracy, Jacobian Frobenius cosine, and the proportion of neuron features in the dictionary and in activations.
    • Judges and counts: sort eval and distress filtering use Claude Haiku 4.5; cluster discrimination uses Claude Sonnet 4.6. Sort eval is 500 times per configuration. The width sweep uses n=2 seeds per cell. The paper does not give one unified repetition count for all experiments.
    • Data: 113 binary classification tasks; 1,500 FineWeb phrase bigrams; replacement evaluation uses FineWeb; training and sort eval cache use SmolLM2. Attribution graphs come from filtered Pile-10k prompts, at most 200 per set, and 90 for Gemma 3 27B and Qwen 3 14B.
    • Thresholds: Table 4 and Appendix J use encoder and decoder cosines both greater than 0.6; Figure 1 and some claims use 0.7.

    Main results

    The table below is based on Table 4: FineWeb, layer depth 66%, cosine threshold 0.6. GS-big (1B, 4B), Gemma 3 12B GS-small, Qwen 3 4B and 8B, and LXTC-32x are omitted. The row with the lowest activation proportion is already included.

    Model and transcoderNF in dictionaryNF in activations
    Gemma 3 1B GS-small5.9%57.9%
    Gemma 3 4B GS-small8.9%63.8%
    Gemma 3 27B GS-small37.1%82.8%
    Qwen 3 1.7B mwhanna4.5%47.1%
    Qwen 3 14B mwhanna11.8%57.7%
    Llama 3.1 8B LXTC-8x39.4%64.9%
    Llama 3.2 1B mntss (cross-layer)13.0%38.0%
    Gemma 2 2B mntss (cross-layer)28.7%49.3%
    • The authors state in Figure 1 that more than 50% of activations in late layers come from neuron features. The table has rows below 50%, including Qwen 3 1.7B and the two cross-layer transcoders. The authors say this claim does not strictly hold on Qwen, and say those transcoders are weaker.
    • The authors say neuron features are still a minority in the dictionary, but are activated more frequently, so their activation share is higher. Figure 6 labels on Gemma 3 4B, Gemmascope small, L22: NF average frequency 1.2e-03, non-NF 7.2e-05, with the title writing 16.9x.
    • The authors say that after subtracting neuron features, the remaining dictionary entries vary more across seeds, do not match the MLP's Jacobian, and move less relative to initialization.

    Replacement, reconstruction, and concept probing

    • The authors reproduce that uncorrected pruned neurons on Gemma 3 4B have high KL and FVU above 1. After adding a linear mapping acting on the pruned output, KL is lower than transcoders, while FVU is still higher. Skip and rank-1 reduce FVU but do not reduce KL (Figure 3a). Appendix A says this KL pattern repeats across multiple models. Figure 12 has no endpoint numbers that can be extracted.
    • Table 5 on eight models, k=1, 113 tasks: the average AUC of neurons is not lower than transcoders in all cases. For example, Gemma 2 2B is 0.851 vs 0.780 (Δ +0.071±0.012), and Llama 3.1 8B is 0.856 vs 0.845 (Δ +0.011±0.007). In Table 6, after taking the peak by layer, at k=1 neurons still win more tasks, with Gemma 2 2B at 87 vs 8, and 18 ties.
    • Table 7 top-1 hits: on Gemma 3 4B, Gemmascope is 75.6%, and sign-split neurons are 0.0%; for Qwen 3 4B, Matryoshka TopK is 21.7%, neurons 0.6%, and ordinary TopK 2.6%. For Gemma 4 26B-A4B, routed-expert neurons are 20.1%, and neurons without routing restriction are 0.9%. The authors say the average AUC of neurons on this extended set is 0.988, so AUC alone makes it hard to judge whether bigrams that neurons do not have were found. On sort eval, the authors say the two are comparable at the top of the distribution, and under quantile sampling neurons' scores drop more noticeably (Figure 3b); the paper does not give readings for each point.

    Alignment, cross-seed, and initialization

    • The authors say that on Gemma 3 4B L22, the Jacobian alignment of all features with the MLP is ≈0.3, and summing only neuron features also reaches this level, while the non-aligned subset is ≈0.05 (Figure 8 main text). In Figure 7, late-layer top-k neurons, without learned correction, also match or exceed the full transcoder. On the two open-source cross-layer transcoders, cross-layer decoder usage is negatively correlated with alignment to the nearest neuron, with Pearson ρ from −0.59 to −0.93 (Appendix Q). The authors say this suggests cross-layer connections are not faithful, and say the evidence is preliminary.
    • Table 1 is Gemma 3 4B L22, TopK, k=64. The authors say that for most configurations, features shared across seeds have more than a 50% chance of being a neuron feature. The 1M-wide shared features are 22.0k, which the authors compare with 20,480 neurons counted by sign. JumpReLU at 262k width has 13k neuron features and 67k cross-seed stable features; the authors say many of these are reproduced items that are not neuron features, and give no mechanism explanation (Appendix D.3).
    • In the TopK pipeline, the median cosine of non-neuron features with the initialization encoder is +0.48, and for neuron features 0.03 (Figure 9; 262k, k=64, 15000 steps). NI on the same model L22, 256k width raises the proportion with encoder similarity greater than 0.5 from 11% to 92%, and the decoder side from 12% to 93%, and the activation mass of aligned features from 65% to 95%.

    Circuits and cases

    • Appendix J: except for Llama 3.2 1B mntss, neuron features account for at least 40% of nodes in the late one-third of layers. In Table 12, for Gemma 3 4B small's late layers, when keeping L0=10, 25, 100 per token, the values are 60±1, 56±1, 60±1. Early layers do not exceed 33%, and drop further under tighter pruning. Figure 15's legend labels neurons (RelP, k=128) 0.64, TopK (k=64) 0.55, Gemmascope L0-small 0.56, L0-big 0.74. The authors say that when L0 is 10 to 100, Gemmascope rather than TopK has a Replacement Score higher than the sparsified MLP; the full MLP is 1.0 at the limit.
    • Line breaks: Gemma 3 4B L20, Gemmascope L0-small, 7 of 8 clean peak-shaped features track a single neuron's character-position curve; another 1 is similar only on the curve, with weights not aligned (Figure 10).
    • Arithmetic: Llama-3.1-8B layer 18, three dictionaries select 50, 48, and 87 features in the write subspace. Ablating all 87, addition accuracy drops from 94.3% to 30.6%; zeroing 28 neurons drops it from 94.3% to 23.6%. High features alone drop it to 75.2%; the authors say mid features have no noticeable effect at steering strengths 1 to 4 (Appendix K). After distress steering in Appendix L, the score drops from 5.1 to 3.9, and the proportion with score not below 5 drops from 66% to 31%; TruthfulQA and MMLU are unchanged to three decimal places.

    Other ablations and analyses

    • Table 10, Gemma 3 4B L22, |cos|≥0.7: matched neurons at 32k, 256k, 1M are 78.3%, 93.3%, 93.0%.
    • Table 14, same layer, authors' TopK: k=64 has FVU 0.166 and KL 0.77; NI with k=32 has FVU 0.209 and KL 1.16. The paper notes that this table's FVU has a different reconstruction target from Gemmascope's.
    • Table 13 absorption rate p1 (neuron top-K=128): Baseline k=32 is 0.64, NI is 0.66, SNRL is 0.71, NFT is 1.00, Gemmascope L0-small is 0.63.
    • Table 15: k-means (K=16) top-neuron matching accuracy is 0.764±0.027, NFT is 0.598±0.033.
    • Appendix O, Qwen 3.5 35B-A3B layer 26, TopK k=32: NF accounts for 15.8% of the dictionary under random initialization, and 30.7% under NI; represented neurons are 25.6% and 44.0%.
    • Under the tested settings, NFT raises FVU by 0.026; SNRL's FVU is 0.018 higher than baseline, its KL is 1.38, and the baseline k=32 in the same table has KL 1.29.
  5. What are some points that can be further explored?

    The authors write that results are limited to base models and English data, dense per-layer transcoders, and only analyze high-magnitude single-token activations.

    Limitations noted by the authors and future directions

    • Results are limited to base models, and mostly use English data (Section 6).
    • The 13 transcoders cover 1B–27B, but are per-layer transcoders on dense models. MoE and cross-layer are only preliminary results, and the authors state that techniques usable for MoE need to be developed (Section 6).
    • Only sparse, high-magnitude, single-token activations are characterized; low activations and mechanisms spanning multiple tokens are beyond this paper's scope (Section 6).
    • Only Top-k and JumpReLU in two codebases are studied. Proportions depend on architecture and on thresholds of 0.6, 0.7, or 0.9; the authors state that they do not give a complete explanation of architectural factors, and do not quantify features that are close to neuron features or more like sparse combinations of neurons (Section 6).
    • Cross-layer analysis uses only two narrow models; faithfulness uses only a first-order Jacobian, with no interventions, and is not extended to non-JumpReLU transcoders (Section 6).
    • There is no systematic human evaluation; quantitative interpretability all uses automated metrics. The authors state that sparse probing may select features that are only correlated with the target variable while the highest activation is another concept. The Discussion also states that whether features such as bigrams and colors that are not in the original basis are truly represented by the original model remains uncertain (Section 6, Discussion).

    Experimental coverage

    • There are 13 open-source transcoders, with model families including Gemma, Qwen, Llama; the authors additionally train TopK and JumpReLU, sweep width up to 1M, and the seed sweep grid is n=2 (Table 4, Appendices A, D.3).
    • Sparse probing uses 113 binary classification tasks; the bigram task uses 1,500 FineWeb phrases, and checks whether the top-1 highest-activation example contains the target phrase (Appendix B, Table 7).
    • Replacement and reconstruction report FVU and KL on FineWeb. sort eval is 500 times per configuration, judged by Claude Haiku 4.5. Attribution graphs are at most 200 per set, of which Gemma 3 27B and Qwen 3 14B are 90 (Appendices B.3, J.1).
    • The paper states that no systematic human evaluation was done. The judge for clustered sort eval is Claude Sonnet 4.6 (Appendix B.4). The distress filter likewise uses Claude Haiku 4.5 (Appendix L).
    • The paper does not give accuracy readings for each sampling setting of Figure 3b in the main text; the per-model curves of Figure 12 also have no extractable endpoint numbers.
  6. Summarize the main content of the paper

    The authors say sparsified neurons can do replacement better, while transcoders' late-layer activations mostly come from neuron features.

    The authors compare sparsified gated MLP neurons with transcoders, and ask what the latter additionally contribute to circuit analysis.

    Polysemanticity is treated as an obstacle to directly analyzing MLPs, so attribution graphs are built on learned dictionaries. The results of Arora et al. and Ye et al. make whether neurons must be replaced a question. The authors define dictionary entries strongly aligned with a single neuron as neuron features.

    The approach is to split each gated neuron by activation sign into one effective encoder and one decoder, then do per-layer top-k. If a feature exceeds the cosine threshold on both sides, it is recorded as a neuron feature. Then Jacobian, cross-seed reproduction, and movement relative to initialization are compared.

    The authors' main reported results:

    • After linear correction, the replacement KL of pruned neurons is lower than transcoders, while FVU is still higher.
    • Table 4 on FineWeb, 66% layer depth, cosine greater than 0.6: in Gemma 3 27B GS-small, NF accounts for 82.8% of activations, and the Llama 3.2 1B cross-layer transcoder is 38.0%.
    • On 1500 bigrams, Gemmascope's top-1 hit is 75.6%, and neurons under the same setting are 0.0% (Table 7).
    • On Gemma 3 4B L22, the Jacobian alignment of all features is ≈0.3, and non-neuron features ≈0.05.
    • After ablating 87 addition-related features, addition accuracy drops from 94.3% to 30.6% (Appendix K).

    The authors' conclusion is: neuron features may be the main structure in transcoders that is consistent with the MLP; the non-neuron part contributes little to first-order computation and depends more on the seed. The authors suggest not using transcoders when width expansion is below 4x, and say that for MoE this effectively means not using them. When comparing with the neuron baseline, the authors suggest using a TopK of about 64, and treating positive and negative activations as different units.

Read originalarxiv.org