Generalization or hallucination? Understanding out-of-context reasoning in Transformers

After fine-tuning injects new facts, whether generalization and hallucination come from the same mechanism.

• Scenario: The authors state that a model can learn new facts during fine-tuning and, at test time, infer implications not written in the training text; other work finds that injecting new facts easily leads to factually incorrect answers. The authors believe it is unclear why both coexist.
• Existing gap: Feng et al. [2024] observed that when several subjects appear simultaneously in two events, the model associates the two events, and provided an empirical mechanistic analysis. The authors state that a theory of training dynamics is still missing, and that erroneous inferences caused by non-causal associations have not been placed in the same framework.
• Core claim: The authors believe both are out-of-context reasoning (OCR, also called ripple effects): inferring implications not directly trained on through concept associations. When the association is causal, it manifests as generalization; when there is no causality, it manifests as hallucination.
• Content of this paper: Test this claim on five LLMs using causal and fictional associations; formalize OCR as symbolic fact recall; compare factorized and non-factorized parameterizations of a one-layer single-head attention-only model; explain the difference using the implicit bias of gradient descent on the merged output-value matrix.

In Section 1.1, the authors divide related work into OCR, transformer training dynamics, and implicit bias.

• Out-of-context reasoning: Krasheninnikov et al. [2023], Berglund et al. [2023a], and Allen-Zhu and Li [2023] discuss OCR from out-of-context meta-learning, situational awareness, and knowledge manipulation, respectively. The paper states that negative results were reported on the reversal curse (Berglund et al. [2023b]) and multi-hop in-weight reasoning (Yang et al. [2024], Biran et al. [2024]). Feng et al. [2024] report that when multiple subjects are simultaneously involved in two events, the model associates them, and provide an empirical mechanistic analysis; Peng et al. [2025] measure generalization or hallucination using a linear transformation between the logits of two related pieces of knowledge. Gekhman et al. [2024], Kang et al. [2024], and Sun et al. [2025] report that fine-tuning on new facts easily leads to hallucination.
• Training dynamics: Induction heads, fact recall, the reversal curse, chain of thought, and in-context two-hop reasoning have been analyzed under training dynamics (Boix-Adsera et al. [2023], Nichani et al. [2024a], Zhu et al. [2024], Wen et al. [2024], Guo et al. [2025], etc.). Tarzanagh et al. [2023a] and Sheen et al. [2024] point out that the optimization dynamics of (WK, WQ) and the merged WKQ differ. The authors state they further examine the generalization consequences of this difference, shifting the focus to value and output.
• Implicit bias: Lyu and Li [2019], Vardi et al. [2022], etc., connect gradient descent to margin maximization. The paper states that the closest works are Tarzanagh et al. [2023a] and Sheen et al. [2024]: gradient descent implicitly minimizes the nuclear norm of the merged query-key weights. Zhang et al. [2025] compare the optimization trajectories of two parameterizations in in-context learning; implicit bias in matrix factorization comes from Gunasekar et al. [2017], Arora et al. [2019], etc. The authors state they focus on value/output and study how this bias affects generalization.

The comparison in the paper is between the factorized (WO, WV) and the merged WOV. The data protocol for LLMs follows Feng et al. [2024], and the form of the symbolic task follows Nichani et al. [2024b]. The authors position the paper as: theoretically analyzing how transformers learn the OCR observed by Feng et al.; and stating that such hallucination also occurs without causal relationships, thereby extending the observation of Peng et al.

The authors formalize OCR as symbolic fact recall, then compare two parameterizations of a single-layer linear attention. The one that can do OCR uses separate output and value; when merged into WOV, expressiveness can be the same, but the limit solutions of gradient flow differ.

Symbolic task and data

• Triples: Knowledge is written as (s, r, a). There are only relations r1 and r2. Answers are split into fact set A1={bi} and inference set A2={ci}, each with n elements, with bi and ci in one-to-one correspondence: if a*(s, r1)=bi, then a*(s, r2)=ci. Multiple subjects can share the same answer.
• Split: Subjects are divided into Strain and Stest. Training includes facts and inferences for training subjects, and facts for test subjects; testing only includes inferences for test subjects. Then split by answer into n equal-sized subsets, each with mtrain training subjects and mtest test subjects.
• Sequence: Form is [s, r, <EOS>, a*(s, r)], predicting the last token from the preceding ones. The vocabulary is subjects, relations, answers, and <EOS>.

Two single-layer parameterizations

• Model: A single layer of linear attention, with hidden dimension theoretically equal to vocabulary size, d=M. The factorized model is f_θ(X)=W_O W_V^⊤ X^⊤ X W_KQ x_T, i.e., predicting the next token with separate output, value, and merged key-query. The non-factorized model merges output and value into WOV=WO WV^T.
• Loss: Next-token probability is softmax over the answer set A only. Training loss is cross-entropy on Dtrain, optimization written as gradient flow; test loss is the same loss on Dtest.
• Expressiveness: Proposition 1 states that when dh≥d, the two parameterizations can achieve the same training and test losses. The authors aim to separate optimization dynamics, not representable functions.

Implicit bias under fixed attention

• Simplification: First assume subjects and relations receive the same attention. Trainable parameters become (WO, WV) or WOV. Given (s, r), the logit is the sum of weights of the answer on the subject column and the relation column.
• SVM: Theorem 1, under regularization and separability assumptions, connects the limit direction of cross-entropy training to an SVM with margin at least 1. The factorized case corresponds to nuclear norm, the non-factorized to Frobenius norm. The authors attribute this to implicit bias of gradient descent.
• Margin: Theorem 2 requires n>1. Under regular conditions, the factorized solution satisfies h_(s,r),a′(W_OV^F)≥min{√(m_train/m_test),1} for test samples, i.e., the test margin lower bound depends only on the ratio mtrain/mtest. The non-factorized solution has margin 0 on test inferences for incorrect answers in A2. The authors state the former has OCR, the latter does not.
• Sample condition: The authors state that as long as mtrain>0, the factorized margin is greater than 0. Thus, a small number of subjects that observe both facts and inferences is sufficient to establish the association, regardless of whether it is causal.

Trainable key-query

• Notation and initialization: WOV(a, z) is the element of the merged matrix at answer a and token z; WKQ(z) is the weight of z for <EOS>. Assumption 2 initializes all WOV(a, z) to α>0, and WKQ(z) to α times the square root of |A|+1.
• Non-factorized: Theorem 3 states that if |A2|>1, the non-factorized model has test loss at least log|A2| at any time, hence greater than 0. The argument uses block symmetry: at test time, probabilities of answers in A2 are equal.
• Excluded cases: The authors state that the factorized model with trainable WKQ exhibits higher-order parameter interactions, and this analysis is left for future work. LLM experiments use AdamW, not the gradient flow of this section.

Experimental setup

• LLMs: Gemma-2-9B, OLMo-7B, Qwen-2-7B, Mistral-7B-v0.3, Llama-3-8B. Appendix E uses Claude-3.5-sonnet to generate 5 examples for some topics.
• Synthetic data: 100 fictional names, with 5 facts paired with 5 implications. The five associations are City–Language, City–Language (CF), Country–Code, Profession–Color, Sport–Music. The first uses real causality, which the authors view as generalization; the latter four are counterfactual or fictional pairings, which the authors view as hallucination. S is divided into 5 subsets, training proportion 0.2, with 4 training subjects per subset.
• Training and metrics: AdamW, full batch, 100 epochs. The learning rate is swept over 10^-6, 3×10^-6, 10^-5, 3×10^-5, 10^-4, 3×10^-4, and the one with the lowest test rank is reported. Mean-rank is the average rank of the paired implication among A1∪A2 by predicted probability, with Rank 0 being the most probable token; lower means the pairing is pushed further to the front. 3 random seeds, with standard errors in parentheses in Table 1. No LLM evaluation.
• One-layer model: orthogonal one-hot embeddings (Section 3.2 says orthogonal embeddings, Appendix E says one-hot), |S|=80, n=20, m=4, mtrain=1, d=dh=128. Trained with cross-entropy for 2×10^4 steps, learning rate 5×10^-4. The comparison is a non-factorized model on the same task.
• PopQA-OCR: Appendix D takes birthplace as the fact and sport as the implication from PopQA, randomly samples 500 subjects, randomly pairs 10, training proportion 0.5, 3 runs. The authors state the training samples are about 10 times the synthetic set, in question-answer format, with real person names as subjects.

Main results

The pairings in the hallucination columns are fictional associations specified in training, not external world knowledge. A low mean-rank means the model ranks this pairing near the front.

Swipe horizontally to read the table

Model	City–Language	City–Language (CF)	Country–Code	Profession–Color	Sport–Music
Gemma-2-9B	0.00 (0.00)	0.19 (0.20)	0.19 (0.07)	1.64 (0.01)	0.56 (0.01)
OLMo-7B	0.07 (0.03)	1.33 (0.49)	0.15 (0.13)	1.84 (0.23)	0.17 (0.01)
Qwen-2-7B	0.13 (0.01)	4.55 (2.33)	3.63 (1.10)	0.82 (0.34)	0.40 (0.08)
Mistral-7B-v0.3	0.00 (0.00)	2.10 (0.01)	1.48 (0.52)	1.15 (0.56)	1.28 (0.13)
Llama-3-8B	0.00 (0.00)	1.18 (0.61)	0.77 (0.10)	0.93 (0.21)	0.63 (0.22)

• Authors' interpretation: The authors state that when causally related, what they call strong generalization appears, consistent with Feng et al. [2024]; without causality, the model still tends to learn the association, which they call a double-edged sword. Four training subjects per subset suffice, and the authors take this as evidence that the two behaviors may come from the same mechanism.
• Relation to pretraining: The authors believe the generalization column is better, possibly because newly injected causal knowledge is consistent with pretraining knowledge and easier to learn.
• Differences within the table: The four hallucination columns are not all close to 0. OLMo-7B's Country–Code is 0.15, and Qwen-2-7B's City–Language (CF) is 4.55 (Table 1). The paper does not explain this difference across models.

Real data PopQA-OCR

• What was tested: Birthplace and sport are randomly paired, then split into training/test as in Section 2, with the metric still mean-rank.
• Results: Table 2 gives Gemma-2-9B 0.68 (0.27), OLMo-7B 1.41 (0.29), Qwen-2-7B 3.30 (0.62), Mistral-7B-v0.3 1.36 (0.70), Llama-3-8B 2.29 (4.26).
• Authors' interpretation: The authors state that OCR-driven hallucination still occurs on real data. Llama-3-8B's standard error is 4.26, and the paper does not explain the dispersion across runs.

One-layer attention

• What was tested: Factorized versus non-factorized on the same symbolic task, with the setup given in the experimental setup. Both models reach a training loss of 0.
• Results: Only the factorized model reaches a test loss of 0; the non-factorized model cannot generalize to test implications (Section 3.2). Figure 3 only shows loss curves, and the text does not list the curve endpoints.
• Authors' interpretation: The caption of Figure 2 states that the factorized model learns a weight structure that makes OCR possible, while the non-factorized one does not; the figure shows the reduced matrix relevant to prediction, and the task is to predict c2 when (s2, r2, c2) is not in the training set. The text states that the non-factorized model's test-implication block weights are zero, while the factorized model has similar patterns on the training and test blocks.

Other ablations and analyses

• Intrinsic dimension: The text sweeps dh∈{3, 4, 8, 16, 32, 128}, with d=128 and mtrain=3; the legend of Figure 6 also includes 64. The authors state that OCR still appears with dh as small as 4. The loss endpoints for each dh are not given in the text.
• Numerical SVM solutions: |S|=12, n=3, m=4, mtrain=3. The caption of Figure 5 states the solution matches Figure 2; after retaining singular values greater than 10^-5, dh=3, and the cosine similarity of the relevant rows is greater than 0.9.
• Learning rate: Appendix E only reports the result with the lowest test rank among six learning rates; the paper does not report mean-rank for each learning rate.
• One-layer repetitions: The paper does not report the number of repetitions for the one-layer loss curves. Tables 1 and 2 for the LLMs use 3 runs.

Limitations and future directions noted by the authors

• One-layer theory: Section 5 and the NeurIPS checklist state that the theoretical results only address a one-layer transformer, and the authors believe extending to multiple layers would be interesting.
• Suppressing hallucination: Section 5 lists "how to prevent such hallucination when injecting new facts" as a future direction. The paper does not provide this method.
• Trainable WKQ: Section 4.2 states that extending Theorem 3 to a factorized model with trainable WKQ would introduce higher-order interactions, and the authors leave this analysis as future work.
• Multi-class convergence: Appendix Remark 2 conjectures that in the multi-class setting, gradient flow also directionally converges to the global minimum of the nuclear-norm SVM, and the authors leave the proof as future work.

Experimental coverage

• LLMs: Section 2 fine-tunes and reports on 5 models: Gemma-2-9B, OLMo-7B, Qwen-2-7B, Mistral-7B-v0.3, and Llama-3-8B.
• Synthetic protocol: five associations, 100 fictional names, training proportion 0.2, 4 training subjects per subset; mean-rank, 3 seeds; among six learning rates, only the one with the lowest test rank is retained (Section 2, Appendix E). The paper does not report mean-rank for each learning rate.
• Real data: PopQA-OCR in Appendix D, 500 subjects, 10 random pairings, training proportion 0.5, 3 runs (Table 2).
• Symbolic model: Section 3.2 and Appendix C compare factorized and non-factorized one-layer single-head linear attention and give numerical solutions for the nuclear-norm/Frobenius SVM. The paper does not report the number of repetitions for this one-layer experiment.
• Optimization objective: The theory in Section 4 concerns gradient flow on cross-entropy or gradient descent with sufficiently small learning rate; the LLM experiments in Appendix E use AdamW, full batch, and 100 epochs.

The authors attribute both generalization and hallucination after fine-tuning new facts to OCR: the model infers implications not written in the training text based on concept co-occurrence, without requiring that co-occurrence to be causal.

• Question: Whether these two behaviors are the same mechanism. In the LLM experiments, some subjects see both the fact and the implication, while test subjects see only the fact and are then asked about the implication. City–Language uses real causality; the other four are fictional or counterfactual pairings.
• Symbolic model: In one-layer single-head linear attention, separate output and value and the merged WOV can have the same expressiveness. The authors state that the limit directions of gradient flow correspond to margin maximization under the nuclear norm and the Frobenius norm, respectively, and only the nuclear-norm solution leaves a nonzero margin on test implications, with the lower bound depending only on the ratio of mtrain to mtest.
• LLM numbers: In Table 1, the City–Language mean-rank is 0.00 (Gemma-2-9B, Mistral-7B-v0.3, Llama-3-8B), 0.07 (OLMo-7B), and 0.13 (Qwen-2-7B). Fictional associations are also learned, but to varying degrees: OLMo-7B's Country–Code is 0.15, and Qwen-2-7B's City–Language (CF) is 4.55. On PopQA-OCR, Gemma-2-9B is 0.68 and Qwen-2-7B is 3.30 (Table 2).
• One-layer numbers: Training loss reaches 0 for both parameterizations; only the factorized model reaches a test loss of 0 (|S|=80, n=20, mtrain=1).
• Authors' conclusion: Co-occurrence alone is sufficient for the model to associate facts with implications, so hallucination occurs when injecting co-occurrences inconsistent with causality. The authors leave the theory for multiple layers, and how to avoid such hallucination when injecting new knowledge, as future work.