GPT-2-Small, Gemma-2-2B, and Llama-3.1-8B have interpretable narrative mechanisms

Whether language models internally contain components related to narrativity that change behavior when intervened upon.

• Problem scenario: The authors claim that when models write stories, they rely on an internal expectation about story structure, i.e., an implicit narrative theory. Whether narratological concepts correspond to consequential distinctions inside the model remains an open question, the authors argue.
• Existing gap: Prior work has described narrative patterns in pretraining data and generation, or used linear probes to detect narrative incoherence. The authors claim that how internal mechanisms causally contribute to narrative behavior has not been examined.
• Operationalization: Following Herman (2009), the authors define narrativity as the mean of three 1–5 ratings—agency, event sequencing, and world building—and search for relevant representations accordingly.
• This paper's analysis: In each layer of GPT-2-Small, Gemma-2-2B, and Llama-3.1-8B, they use Narrativity Value Correlation (NVC) to identify attention heads, MLP neurons, and residual stream SAE features, then ablate the sets.
• Questions to answer: Which narrative representations are used at inference time; whether they align with narratological dimensions; and whether narrative mechanisms are used for the intelligent activities Herman (2013) describes, such as commonsense reasoning and confabulation.

Computational Narratology and Narrative Tasks

• Piper and Bagga (2022): Turn the formal dimensions of Genette (1980) and Herman (2009) into linguistic features, combined with lexical and syntactic n-grams, and use logistic regression, support vector machines, and random forests to predict narrativity. The paper relays their finding that certain patterns matter more to human perception of narrativity, and proposes the distant worlds theory.
• Narrative in Data and Generation: Johnson et al. (2026) characterize narrative content in pretraining data along nine dimensions, stating that the dimensions that make cognitive experientiality possible vary across sources. Hamilton and Mimno (2026) and Tian et al. (2024) report that generated story content is highly repetitive and more homogeneous than human samples.
• Narrative Benchmarks and Hallucination Hypotheses: StoryCloze (Mostafazadeh et al., 2016) and CosmosQA (Huang et al., 2019) are written as narrative benchmarks testing commonsense, causal, and contextual understanding. Sui et al. (2024) cite Herman (2013) and hypothesize that language model confabulation/hallucination is a form of narrative expression.

Internal Narrative Representations

• Probes and Embeddings: De Langis et al. (2026) use linear probes to detect narrative incoherence at different layers; Volpetti et al. (2020) and Hatzel and Biemann (2024) use encoder embeddings to represent characters and plots.
• Authors' Positioning: The authors state that this work shows internal narrative representations exist, but the causal mechanisms affecting output, and whether they constitute a resource for downstream tasks, have not been empirically tested.

Component Interventions

• Positioning of Language Variants: Liu et al. (2026b), Tang et al. (2024), and Deng et al. (2025) intervene on attention heads, neurons, and SAE features related to language variants, respectively. The authors state that this work shows language behavior can be localized within components and causally manipulated.
• SAE and Interpretability: Huben et al. (2024) use sparse autoencoders to decompose activations; the explanations used in this paper come from Neuronpedia (Lin, 2023), pre-generated by the unsupervised method of Bills et al. (2023).

Baselines and Data

• Baseline Methods: Random ablation of equal-size components. The paper cites Yamamoto et al. (2026) and Wang et al. (2026) as practices for random ablation controls in mechanistic interpretability.
• Benchmarks/Datasets: Identification uses NarraDetect (Piper and Bagga, 2025). Downstream tasks are XStoryCloze, CosmosQA, OpenBookQA, TruthfulQA, FaithDial, BEGIN.

The authors state that prior computational narratology studied the phenomena described by Herman, but lacked causal tests of internal narrative mechanisms; this paper, beyond representation probing, intervenes on narrativity-related components.

The overall approach is to use narrativity value correlation (NVC) as the selection criterion, find NarraHeads, NarraNeurons, and NarraFeatures at each layer, ablate them as a group, and compare with equal-size random ablation. Causal claims target the sets, not individual units.

Narrativity and Component Contributions

• Three-Dimensional Definition: The main text writes narrativity as the mean of three items. Agency emphasizes actors who are continuously present and experience events; event sequencing emphasizes the temporal organization of events; world building emphasizes an experienceable, coherent story world. The scoring rubric is in Appendix A, where the item is written as World Making.
• Contribution Values: For attention heads, take the L2 norm of the token's output vector (Kobayashi et al., 2020); for neurons, take their activation; for SAE features, take the feature activation of the pretrained autoencoder on the residual stream. Text-level contribution is the average over content tokens, excluding BOS. The authors state that BOS is an attention sink, and ablation does not change the BOS position.
• Form: Narrativity=mean(Agency,Event,World), i.e., the mean of three 1–5 scores. For a given text x, n is narrativity, c is the token-averaged contribution of the component, and ΔL is the increase in average next-token loss after ablating the set.

Selecting Components with NVC

• Two Criteria: NVCcont = corr(c, n), NVCloss = corr(ΔL, n). The authors use Spearman, reasoning that it does not assume linearity, only monotonicity. They also compute NVCcont(a), NVCcont(e), and NVCcont(w) for each of the three dimensions.
• Selection Rule: Use NVCcont ≥ 0.5 as a heuristic threshold (Cohen, 2013). Selection does not depend on NVCloss; NVCloss is used for subsequent ablation validation. The authors state they make no prior assumptions about how contributions manifest in downstream tasks.
• Data Split: Identify components on the expanded NarraDetect silver labels (n=13,530, 18 genres, median word count 104); human gold labels n=394 are held out for Section 5.1 validation. Silver labels are averaged from Gemma-4-31B, GLM-4-32B, and Qwen3.5-27B, and the main text states Spearman ρ ≥ 0.795 with human labels.

Ablation Implementation

• Three Interventions: Zero out attention head outputs; zero out selected neuron activations, thereby removing the contribution of their corresponding output rows; for SAE, do not replace the residual stream with a reconstruction, but subtract the decoder contribution of the selected features.
• SAE Intervention Form: x^abl_t=x_t−∑_{j∈ S}z_t,jd_j. The authors state this preserves residual components not reconstructed by the SAE. GPT, Gemma, and Llama use per-layer residual stream SAEs: GPT-2 uses TopK (k=32, 32k features), Gemma Scope uses the canonical SAE with mean L0 closest to 100, and Llama Scope has 32k features with 8x expansion.
• Random Control: Each target ablation is paired with an equal number of random sets, seeds 0–9, sampled from the full component pool of that model or SAE, without excluding target components, and results are averaged. The authors state that ablating individual components one by one is computationally infeasible, so causal statements are made only about sets.

Feature Classification and Task Determination

• Dimension Classification: Each NarraFeature is assigned to the dimension with the highest NVCcont. Then test whether Neuronpedia's natural language explanations alone can recover this classification: all-MiniLM-L6-v2 embeddings with three-class logistic regression, and DeepSeek-V4.1-Flash zero-shot. The classifier is trained on explanations from two models and tested on a third (Appendix F). Explanations were automatically generated before the experiments, on other data.
• Open Generation: 900 prompts, 450 pairs, differing only in the requested genre (story or explanation); 90 topics, five instruction templates. Decoding is contrastive search, α=0.6, k=4, at most 128 new tokens. Narrativity is scored by DeepSeek-V4.1-Flash and HY-3, and the main text states ρ ≥ 0.83 with NarraDetect human labels. GPT does not enter the generation experiments; the authors write that it cannot stably produce coherent output.
• Downstream Scoring: Six benchmarks zero-shot, using the model's own next-token log probabilities, without chat templates and without generating text (Appendix H). XStoryCloze uses ℓ/T at the ending; multiple-choice tasks read the log probability of the option letter; three hallucination benchmarks use ℓ/B(y). GPT does not enter the benchmark experiments; the authors write that its baselines are near random and it lacks multilingual ability.

实验设置

• 被测模型：GPT-2-Small-124M、Gemma-2-2B、Llama-3.1-8B，均为预训练基座。生成与基准实验只有后两者。SAE分别为OpenAI的GPT-2自编码器、Gemma Scope、Llama Scope。
• 识别与验证数据：银标签NarraDetect 13,530篇用于按NVCcont ≥ 0.5选组件；人类金标签394篇用于NVCloss。开放生成900条提示。
• 下游基准与规模：XStoryCloze英语、俄语、中文、印地语，各1,511篇；CosmosQA验证集2,985；OpenBookQA测试集500；TruthfulQA 817；FaithDial 2,924对；BEGIN 676个上下文（Appendix H）。
• 对照与指标：等规模随机消融，10个种子。指标为Spearman NVCcont、NVCloss、1–5叙事性、准确率，以及ΔTarget−ΔRand（百分点）。
• 评审：组件识别的银标签由三个开源模型评分；生成叙事性由DeepSeek-V4.1-Flash与HY-3评分。基准不是LLM裁判，而是对数概率。机会水平：XStoryCloze 50%，两个QA为25%，TruthfulQA 22.6%，FaithDial 50%，BEGIN 45.6%。
• 检验：配对bootstrap、双侧、p < .05、未校正。论文写明随机结果对种子0–9平均。论文未报告bootstrap重抽样次数。

主结果

Table 1报告消融后损失升幅与叙事性的Spearman相关。下表为点估计；95% CI见该表。省略Gemma-2-2B的NarraHead（n=3，1.4%，NVCloss .13，随机−.06）。

Swipe horizontally to read the table

模型	组件	n（占比）	NVCloss	随机
GPT-2-Small-124M	Head	2（1.4%）	−.01	.06
GPT-2-Small-124M	Neuron	2534（6.9%）	.84	−.04
GPT-2-Small-124M	Feature	695（0.2%）	.83	.01
Gemma-2-2B	Neuron	3470（1.4%）	.65	.02
Gemma-2-2B	Feature	2529（0.6%）	.86	.07
Llama-3.1-8B	Head	28（2.7%）	.27	−.06
Llama-3.1-8B	Neuron	4157（0.9%）	.65	.01
Llama-3.1-8B	Feature	1211（0.1%）	.84	.04

• 选择性：作者称随机消融在不同叙事性文本上的损失增加相近；按NVCcont选出的集合在8/9个设置中随叙事性变化，唯一例外是GPT的2个NarraHeads。Figure 4标注Llama上头、神经元、SAE特征的NVCloss为0.27、0.65、0.84。
• 组件类型：作者称注意力头在三个模型上都是效应最弱的类型；神经元和SAE特征对高叙事性文本的损失影响大于低叙事性文本。
• 解释边界：作者在Discussion 8.1写明，较大的SAE消融效应本身并不能确立残差流表征比注意力头或神经元更重要。

开放生成

• 故事与解释：Table 5中，Gemma故事生成为3.80/3.15/3.13，解释为2.17/1.78/2.24；Llama故事为4.05/3.27/3.28，解释为1.78/1.49/1.96（Agency/Event/World）。Figure 5只报告Llama：故事激活高于解释的比例为头91%、神经元92%、SAE特征94%。论文未给出Gemma的对应比例。
• 整组消融：Table 7中，Gemma的NarraFeature三维变化为−1.86、−1.40、−1.27，随机为−.36、−.44、−.36；Llama为−1.53、−1.20、−1.03，随机为−.06、−.11、−.05。作者报告这些目标与随机的差异为配对bootstrap双侧p < .05（未校正）。神经元降幅小于特征、大于随机；头与随机接近，未加粗。Figure 1标注Llama上450条故事提示的平均叙事性从3.54到1.49，10种子随机消融从3.54到3.39；图中示例从4.7/5到2.0/5。该1.49与Table 7的分维度Δ不是同一汇总，论文未说明二者如何对应。
• 分维度：Table 8的故事条件下，Gemma的agency组为−2.48/−1.96/−1.60，event组为−1.19/−1.55/−0.92，world组为−1.10/−1.17/−0.77；Llama分别为−2.38/−2.02/−1.57、−0.99/−1.59/−0.91、−0.45/−0.64/−0.44。解释条件下降幅更小，Llama的world组为−0.07/−0.06/−0.03。作者称agency与event组在对应维度上的原始降幅最大，但干预影响多个维度，且agency组对event的降幅大于event组；未观察到world building的选择性靶向效应。作者将其解释为部分对齐，而不是可独立控制的维度。

下游基准

Table 10的数字是相对等规模随机消融的ΔTarget−ΔRand（百分点）。加粗规则只用于NarraFeatures。基准准确率：Gemma为82.7、74.7、73.3、63.7、42.6、69.4、25.0、67.6、85.5；Llama为85.0、76.2、76.4、68.0、81.2、86.9、28.9、68.0、85.8（顺序同表列）。

基准	Gemma	Llama
XStoryCloze EN	−9.1±2.3	−6.4±1.9
XStoryCloze RU	−7.6±2.3	−0.7±1.7
XStoryCloze ZH	−5.0±2.0	+0.2±1.3
XStoryCloze HI	−5.3±1.7	−1.2±1.4
CosmosQA	−2.8±2.2	−1.6±0.9
OpenBookQA	+2.6±2.5	−0.3±0.4
TruthfulQA	+2.8±2.2	+2.5±2.2
FaithDial	−2.0±1.2	−0.7±1.1

• 叙事任务与对照：作者报告Gemma四种语言的XStoryCloze，以及两个模型的CosmosQA，NarraFeature相对随机为负且p < .05。Llama的俄语、中文、印地语未被标出。BEGIN的相对差为Gemma +0.5±1.6、Llama +0.5±1.4，未加粗。作者称OpenBookQA上Gemma的降幅小于随机、Llama与随机相近。Table 16给出绝对变化：Gemma英语XStoryCloze的ΔTarget为−11.6、ΔRand为−2.6；Llama为−7.1与−0.7。
• 幻觉：作者称TruthfulQA上NarraFeature消融相对随机提高准确率，并认为这与Sui et al. (2024)把confabulation视为叙事化的假设相符。同一消融在BEGIN上与随机相近；Gemma的FaithDial为−2.0，作者称方向相反，因为该任务要选择忠实回应。作者称幻觉相关基准的结果mixed。
• 头与神经元：Table 10未按同一规则给这两类加粗。作者称其XStoryCloze效应幅度更小且不一致，并称Llama上消融NarraHeads对非英语XStoryCloze的影响大于NarraFeatures。该表中Llama Heads的点估计为EN −1.5、RU −2.8、ZH −1.8、HI −2.2个百分点。

特征解释与维度相关

• 归类比例：Table 2中，GPT的695个特征为Agency 33.7%、Event 29.4%、World 37.0%；Gemma的2529个为40.2%、21.7%、38.1%；Llama的1211个为34.4%、34.2%、31.5%。
• 解释能否恢复维度：成功取回4434/4435条解释。Table 4的Macro-F1：嵌入分类器为GPT .53、Gemma .57、Llama .58；DeepSeek零样本为.50、.47、.48；机会水平.33。作者称解释含有高于机会的维度信号，但噪声大：有的解释描述标点或句末，相近解释也可得到不同维度标签。
• 生成时的相关：作者称归类后的特征贡献与对应维度评分的Spearman ρ ≥ 0.8。Table 6中对应格为Gemma 0.82、0.88、0.89，Llama 0.82、0.89、0.88；95% bootstrap不超过±0.04。非对应格有低于0.8者，如Llama agency特征与World为0.70。

其他消融与分析

• 退化输出：未干预时Gemma 2.1%、Llama 8.1%的生成被标为退化；SAE目标消融为12.7%与14.1%，随机消融中位数为6.7%与7.0%。作者称去掉退化样本后，SAE目标效应变化至多0.11，且仍低于全部10个随机种子（Appendix G、Table 13）。
• 无法评分的生成：22条全部来自Gemma-2-2B，其中21条在SAE特征消融下；作者将其三维都记为1。
• 截断：生成在首次出现指令框续写标记处截断，涉及Gemma生成的15.2%、Llama的5.3%。
• 负相关组件：作者称也观察到与叙事性负相关的组件（ρ ≤ −0.5），但未对其做消融分析。

作者指出的局限与后续方向

• 模型范围：Limitations写明，研究限于三个预训练语言模型。
• 选择与因果单位：Limitations写明，组件用启发式阈值选择；因果主张适用于被消融的组件集合，而不是单个单元。
• SAE效应的解释：Discussion 8.1写明，观察到的较大SAE消融效应本身不能确立残差流表征比注意力头或神经元更重要。同一节称NVC本身可能偏向某些特征或组件，并用熵代替相关、用概率代替取值等做法留待未来工作。
• 反叙事组件：Discussion 8.1称ρ ≤ −0.5的组件超出研究问题范围，调查模型如何表示这种负向关联是未来方向。
• 理论与循环论证：Discussion 8.2称，换用另一套叙事理论（如Sternberg, 2011）也可能得到结果，哪些理论能描述模型内的叙事表征需要更多研究。Discussion 8.3称，用叙事相关任务检验sense-making、再得出叙事表征是sense-making资源，是循环论证；未来工作需要用更多样的方式检验。
• 幻觉指标：Discussion 8.3写明，评测衡量的是固定候选回答的相对对数概率，不是开放生成中的幻觉；似然转向真实或可归因候选，本身不能确立消融在实践中会产生更少幻觉。

实验覆盖范围

• 模型与任务分工：组件识别覆盖GPT-2-Small-124M、Gemma-2-2B、Llama-3.1-8B。生成实验与基准实验按Section 4.3、4.4不纳入GPT；作者写明的原因分别是不能稳定产生连贯输出，以及基线接近随机且缺少多语言能力。实验使用的是非指令微调的基座模型。
• 识别数据：英文NarraDetect银标签13,530篇用于选择，人类金标签394篇用于NVCloss验证（Section 4.1、5.1）。正文给出银标签与人类标签的Spearman ρ ≥ 0.795，生成评审与人类标签ρ ≥ 0.83（Table 11另有分模型数值）。
• 生成协议：Gemma与Llama各对同一900条提示生成；contrastive search的设置为α=0.6、k=4、最多128个新token；叙事性由DeepSeek-V4.1-Flash和HY-3评分（Section 4.3、6）。
• 下游协议：基准为零样本对数概率，含XStoryCloze四种语言、CosmosQA、OpenBookQA、TruthfulQA、FaithDial、BEGIN。随机消融为10个种子。Table 10把Gemma四种语言及两个模型CosmosQA上的NarraFeature相对差标为p < .05，Llama的XStoryCloze只标出英语。检验写明为未校正。
• 论文未报告的项目：论文未报告两位生成评审之间的一致性，也未报告bootstrap重抽样次数。Figure 5的故事对解释激活比例只给出Llama。

The authors connect the narratological three-dimensional definition to mechanistic intervention, examining whether narrativity-related components in pretrained language models participate in prediction, generation, and downstream tasks.

• Problem: Narrativity is defined as the 1–5 mean of agency, event sequencing, and world building. The authors ask which narrative representations are used at inference time, whether they correspond to these three dimensions, and whether they participate in commonsense reasoning and confabulation.
• Method: In each layer of GPT-2-Small-124M, Gemma-2-2B, and Llama-3.1-8B, they use Spearman-correlation NVC (ρ≥0.5) to select attention heads, neurons, and residual stream SAE features, ablate the sets, and compare with equal-sized random ablations across 10 seeds. Identification uses NarraDetect silver labels; validation uses 394 human gold labels.
• Loss: In Table 1, the NVCloss of NarraNeuron and NarraFeature across the three models is on the order of .65 to .86, with random control point estimates near 0; for example, Gemma feature .86 vs. .07, Llama feature .84 vs. .04. GPT's 2 NarraHead is −.01, the exception among the 8/9 settings the authors mention. Attention heads are overall weak.
• Generation: On Gemma and Llama, ablating NarraFeature reduces three-dimensional narrativity more than random (Table 7; Gemma's Agency −1.86, random −.36). Figure 1 annotates Llama's 450 story prompts from 3.54 to 1.49, random 10 seeds to 3.39. Dimension-wise ablation affects more than one dimension; the authors report no selective targeting effect for world building.
• Downstream: Relative to random ablation, Gemma's NarraFeature is lower on XStoryCloze in all four languages (Table 10, English −9.1 percentage points), and Llama mainly marks English (−6.4) and CosmosQA (−1.6). OpenBookQA shows no relative decrease in the same direction. TruthfulQA rises relatively, and FaithDial and BEGIN are inconsistent. The authors claim this provides evidence that the selected sets causally contribute to model behavior, but internal organization, relation to narrative theory, and broader functions remain open questions. The authors plan to release code, generations, review scores, and component lists in a later version.