Multilingual Unlearning in Large Language Models: Transfer, Dynamics, and Reversibility

Core question: In multilingual LLMs, does unlearning performed in one language transfer to other languages, and is the "forgotten" knowledge truly erased.

• Scenario and importance: Data protection regulations require deleting personal information from LLMs, while multilingual models serve multilingual users; the authors argue that if unlearning only acts on a single language surface, an attacker might switch languages to elicit deleted content.
• Gaps in existing research: The authors note that unlearning research is English-centric, with only scattered evidence on cross-lingual transfer; arguments that unlearning is more like suppression than erasure are also made mainly in monolingual settings, and in the multilingual case it is unclear whether it is recoverable, whether it is language-independent, and at which layer it is localized.
• Core hypothesis: If unlearning mainly modulates the language-specific decoding layers of later layers, the underlying knowledge may remain intact and could reappear via cross-lingual prompting or querying in another language; if it changes the shared cross-lingual representation space, the unlearning guarantee is stronger.
• Paper contributions: The authors extend TOFU to five languages and systematically characterize cross-lingual unlearning transfer; they analyze the mechanism using cross-lingual prompting, layer-wise cosine similarity, and PCA; and they construct inference-time steering vectors to recover forgotten knowledge, reporting recovery of 50% on Qwen and 90% on Gemma.
• Threat model (recovery experiments, Section 5):
  • Goal: Increase leakage of forgotten content and recover suppressed knowledge.
  • Knowledge: Knows the forgotten domain but cannot access the original answers.
  • Capability: Has inference-time (white-box) access to the unlearned model and can manipulate internal activations during the forward pass.
  • Victim system: Multilingual LLMs that have knowledge injected via fine-tuning and then undergo unlearning (Qwen2.5-7B, Gemma2-9B).

LLM unlearning methods

• Jang et al. (2023), Zhang et al. (2024), Yoon et al. (2025), etc.—parameter-update unlearning by modifying the fine-tuning objective, including gradient ascent (GA) and preference-style objectives (DPO, NPO); the paper notes that multiple studies consider such methods mechanistically more like suppression and therefore reversible, with recovery usually relying on additional "relearning" or providing an answer prefix.
• Ren et al. (2025)—distinguishes unlearning methods by intent: removing internal knowledge vs. suppressing behavioral expression; the paper uses this to focus on parameter-update methods.
• Li et al. (2024), Huu-Tien et al. (2025), Shen et al. (2026)—representation-level unlearning (representation misdirection), fine-tuning intermediate-layer representations.

Multilingual models and cross-lingual transfer

• Pires et al. (2019), K et al. (2020)—multilingual pre-training produces shared representations that support cross-lingual transfer, but transfer is uneven and stronger for languages of the same type and with shared scripts.
• Lim et al. (2025), Wendler et al. (2024)—mechanistic evidence suggests multilingual LLMs may perform intermediate processing in a shared semantic space aligned with the dominant pre-training language, then map to the target language.

Hidden representations and activation steering

• Wendler et al. (2024), Lim et al. (2025), Jiang et al. (2025)—multilingual transformers exhibit a three-phase structure: input space, a language-independent conceptual space in intermediate layers, and a language-specific decoding space.
• Rimsky et al. (2024), Li et al. (2023)—inference-time activation steering, constructing steering vectors from contrastive activations to control behavior; Seyitoğlu et al. (2024) use steering to retrieve anonymized concepts. The paper's distinction: it treats the representation difference before and after unlearning as a mechanistic suppression direction and uses it as an analysis and intervention tool.

Preliminary evidence on cross-lingual unlearning

• Lu & Koehn (2025), Hwang et al. (2025), Choi et al. (2024)—demonstrated isolated cross-lingual unlearning transfer effects; the authors claim to be the first to give systematic patterns.

Baselines and benchmarks: The unlearning target baselines are GA (Jang et al., 2023) and NPO (Zhang et al., 2024), with DPO as the main method; the evaluation benchmark is TOFU (Maini et al., 2024) and its 1% forget split; the baseline for the recovery experiments is an equal-norm random Gaussian vector. The authors position themselves as differing from the above work in three respects: transfer patterns, layer-wise localization, and relearning-free inference-time recovery.

Overall approach: first extend English TOFU into a multilingual benchmark and define a multilingual unlearning protocol, then run four groups of experiments in sequence: transfer measurement (4.1), cross-lingual prompting (4.2), hidden representation analysis (4.3), and steering vector recovery (4.4).

Multilingual TOFU and notation

• Data extension: TOFU contains 200 fictitious authors with 20 QA items each; the authors used Gemini-flash2.5 to translate the English QA into Chinese, German, Russian, and Turkish, giving L={EN, CH, DE, RU, TU}, with the same forget/retain split shared across languages.
• Language attribute design: English and German share a language family and script, Turkish shares the script but not the language family, Russian shares the language family but not the script, and Chinese differs in both, used to separate the effects of language family and script (footnote 3).
• Unlearning scale: a 1% forget split (|Aforget|/|A| = 0.01) is used, corresponding to a small-sample unlearning scenario that deletes information about only a few individuals.

Fine-tuning and unlearning objectives

• Fine-tuning: the standard autoregressive objective J_FT(θ) = 𝔼_{(x,y)∼ DFT} −log p_θ(y|x), training one checkpoint for each fine-tuning language combination.
• Unlearning: J_UN averages the forget objective over the unlearning language set Lunl and adds a regularization term on the retain set, with weight λ controlling the trade-off; training continues from θFT initialization.
• DPO instantiation: a preference pair is constructed for each forget sample, with the preferred response being an IDK-style refusal dynamically sampled from a pool of 100 language-specific refusal templates and the dispreferred response being the original answer; a KL penalty against a frozen reference model is added (λ=1.0), and each forget sample is randomly paired with 10 retain samples (Appendix B).
• Other objectives: GA and NPO are also tested to cover common unlearning objectives (Appendix D).

Evaluation protocol

• NLI semantic score: xlm-roberta-large-xnli is used to judge whether the generated answer and the reference answer mutually entail each other; the formula includes a symmetric entailment term and contradiction and neutral penalty terms; the authors believe it can capture leakage that differs in wording but is semantically equivalent.
• Transfer metric: reports the change in the forget-set NLI score after unlearning relative to the fine-tuned model, with a larger negative value indicating stronger unlearning; the main experiments are repeated on 5 random forget-author splits, reporting mean ± 95% confidence interval.

Mechanism analysis and recovery method

• Cross-lingual prompting: when the query language is q, the model is instructed to answer in the fine-tuning language ℓ, measuring the performance gain ∆ℓ←q (prompt templates in Appendix G).
• Representation analysis: takes the last-token hidden state at each layer, computes the cosine similarity of representations of the same question across languages, and performs PCA on the forget-set representations; centroid distance and average pairwise distance are also used to quantify the displacement caused by unlearning (Appendix H.3).
• Steering vectors: uses an English auxiliary forget set (obtained by randomly reshuffling retain authors, not the real forget set) to compute the layer-wise normalized representation difference between fun and fft, yielding a direction g(l) for each layer; at inference, α·‖h‖·g(l) is subtracted from fun's hidden state over N+1 consecutive layers. On Qwen, α=0.5 (English, Chinese) or 0.8 (others), N=2; on Gemma, N=6, with α grid-searched by language. α and the injection layers are tuned directly on the forget set, which the authors call an upper-bound estimate of worst-case leakage.

Experimental setup

• Models tested: Qwen2.5-7B and Gemma2-9B; main-text results are mostly Qwen, with Gemma reproduced in Appendices F.2 and J.
• Benchmark: TOFU 1% forget split, extended to EN, CH, DE, RU, TU; main experiments are repeated over 5 random forget author splits (only the unlearned model is retrained), reporting mean ± 95% CI.
• Unlearning methods: main text uses DPO; GA and NPO verify the transfer patterns in Appendix F.3.
• Metric: NLI semantic score (xlm-roberta-large-xnli), reporting the change after unlearning relative to the fine-tuned model, with negative values indicating stronger unlearning.
• Reviewer validation: 4 annotators (native speakers of the non-English languages) annotate 50 items per language, with NLI and human overall agreement of 89.0% (Table 7); GPT-5 reviewer agreement is 94–96%, and ROUGE-L is 62–68% (Table 8).
• Compute: 4 NVIDIA H100 GPUs.

Main results

Table 1 (Qwen2.5-7B, DPO) is the full matrix of 5 fine-tuning languages × 5 unlearning languages × 5 query languages; the table below takes representative cells from each fine-tuning-language block (full values in Table 1):

Swipe horizontally to read the table

FT	Unlearn	Query	Base	∆NLI
EN	EN	EN	93	-90 ± 3
EN	DE	EN	93	-17 ± 7
CH	CH	CH	82	-77 ± 2
DE	EN	DE	90	-41 ± 8
RU	RU	RU	95	-90 ± 7
TU	EN	TU	99	-55 ± 8
TU	TU	TU	99	-95 ± 1

• Both language family and script affect transfer: the authors state that after controlling for script, transfer from English to Russian (same family) is stronger than to Chinese; Turkish, which shares the Latin script but is a different family, transfers more strongly to English and German than to Chinese; when both family and script are shared (English–German), transfer is strongest.
• Transfer asymmetry: the authors state that languages with high pretraining coverage (English, Chinese) are stronger unlearning sources, while low-coverage languages (German, Russian, Turkish) are weaker as sources.
• Unlearning in a weak language still harms a strong language: on the Turkish fine-tuned model, the forget set under English queries is only 11%, yet unlearning in English still lowers the Turkish-query NLI score by 55 (orange cell in Table 1); the authors hypothesize that concepts are accurately represented in a language-agnostic space and that unlearning mainly suppresses later-layer generation.

Cross-lingual prompting (4.2)

• What was measured: under query language q, instructing the model to answer in the fine-tuning language ℓ, measuring the gain ∆ℓ←q (Table 2, Qwen2.5-7B).
• Results: gains are generally positive, e.g., with English fine-tuning, Chinese query +29, German query +61, Russian query +30, Turkish query +27; with German fine-tuning, English query +33.
• Relation to transfer: the Table 2 gains correlate with the Table 1 transfer scores, Pearson r = 0.50 (p < 0.05), Spearman ρ = 0.60 (p < 0.01); the authors interpret this as the stronger the hidden mapping between languages, the greater the harm unlearning one causes to the other.

Hidden representation analysis (4.3)

• Cosine similarity (Figure 1): cross-lingual prompting (Panel 3) mitigates the drop in later-layer similarity under the default setting (Panel 2); the fun and fft curves are close in early-middle layers and diverge only in later layers. The authors state that the performance gap mainly comes from a decoding bottleneck and that unlearning mainly acts on later layers; the authors also note that cosine similarity does not directly measure knowledge retrieval and that the interpretation is limited to representational geometric trends.
• PCA (Figures 2, 6): fbase and fun are separable from the earliest layer; on Chinese questions fft and fun are already separated by layer 10, and at the last layer the three variants each form their own cluster; for English, fft and fun often remain in the same cluster, but the paired representation distance for the same question is still large. The authors interpret this as unlearning pushing representations away from the fine-tuned state but not back to the base distribution, with knowledge reorganized or suppressed rather than removed.
• Distance decomposition (Appendix H.3, Figure 7): the centroid distance between fft and fun is approximately constant across layers (global displacement is stable), while the average pairwise distance rises near the output layer; Chinese shows a centroid-distance spike at about layers 8–18, as an exception.

Steering-vector recovery (4.4)

• Qwen (Figure 3 left): a vector extracted using only the English auxiliary set, injected in middle-to-late layers, recovers over half the lost performance on the English forget set and recovers across languages (most pronounced for Chinese); random Gaussian vectors yield only small and inconsistent gains (Figure 3 right). The abstract gives recovery proportions: Qwen 50%, Gemma 90%.
• Gemma (Appendix J, Figure 8): the authors state that the steering intervention recovers almost all of the fine-tuned model's capability across languages, stronger than Qwen.
• Chinese as source language (Appendix K.1): best recovery scores are English 68.21%, Chinese 49.89%, German 41.67%, Russian 38.21%, Turkish 56.15%.
• Other unlearning methods (Appendix K.2, Gemma2-9B): under GA the best recovery is 41.58%–80.60% (highest for Chinese), under NPO 51.49%–68.95% (highest for English); the authors state that recoverability is not limited to DPO.

Other ablations and analyses

• Multilingual joint fine-tuning (Appendix F.1, Table 9): the main trends are consistent under multilingual fine-tuning + multilingual unlearning combinations, e.g., with five-language fine-tuning and unlearning with ENCHRU, English query -94 ± 2, Chinese -99 ± 1.
• Gemma transfer (Appendix F.2, Table 10): single run, reduced combinations, with consistent script/family and coverage-asymmetry trends.
• GA/NPO transfer (Appendix F.3, Tables 11–12): qualitatively consistent patterns, with absolute magnitudes varying by method.
• Retain set (Appendix F.4, Table 13): cross-lingual retain changes are generally very small with no consistent pattern, in contrast to the forget set.
• Data contamination check (Appendix A, Table 3): base model ROUGE-L Recall is Qwen 0.44, Gemma 0.37, comparable to the TOFU original paper's baseline; the authors state that contamination cannot be fully ruled out.

Limitations and future directions noted by the authors

• Limited algorithm coverage (Section 5): The authors do not claim the conclusions apply to all unlearning methods, and leave comparison across more algorithms to future work; they cite Lu & Koehn (2025) as saying the conclusions may extend to more fine-tuning-based unlearning algorithms.
• Task and scale limitations (Section 5): TOFU consists of simple factual queries that do not require complex reasoning, so the conclusions are best interpreted in the context of small-scale factual retrieval and may not transfer to more complex content or larger forgetting proportions.
• Knowledge source limitations (Section 5): TOFU knowledge is injected via post-training, so the conclusions may not fully characterize the multilingual forgetting behavior of knowledge acquired during pre-training; the authors leave systematic study of this distinction to future work.
• Data contamination cannot be fully ruled out (Appendix A): The pre-training corpus is not public, and timeline and baseline score checks cannot completely rule out contamination.

Experimental coverage

• The models tested are Qwen2.5-7B and Gemma2-9B, two in total; the Gemma transfer experiments are a single run with a reduced set of language combinations (Appendix F.2).
• Language coverage is EN, CH, DE, RU, and TU, five in total; forgetting targets cover DPO, GA, and NPO, three in total, with GA/NPO limited to five single-language fine-tuning settings and no repeated splits (Appendix F.3).
• The forgetting scale is the TOFU 1% forget split; the main experiments repeat over 5 random forget splits.
• Evaluation uses NLI automatic scoring, with manual verification of 50 items per language by 4 annotators in total; the paper does not report inter-annotator agreement.
• The α and injection layer of the recovery experiments are tuned directly on the forget set, which the authors position as an upper-bound estimate of worst-case leakage (footnote 12).

An analysis paper on multilingual LLM unlearning, studying whether unlearning performed in one language transfers to other languages and whether forgotten knowledge is truly erased.

• Problem: Unlearning research is English-centric; under multilingual deployment, single-language unlearning evaluation may overestimate robustness, and an attacker may switch languages to elicit deleted content.
• Method: Extend TOFU to five languages EN, CH, DE, RU, TU (1% forget split), systematically permuting fine-tuning, unlearning, and query languages; unlearn with DPO (also testing GA, NPO), evaluate with NLI semantic scores; further conduct cross-lingual prompting, layer-wise cosine similarity and PCA analyses, and use steering vectors extracted from an auxiliary forget set to inject in reverse at inference time.
• Transfer results (Table 1, Qwen2.5-7B): Unlearning transfers across languages, more strongly between languages of the same family or script, and languages with high pretraining coverage are stronger unlearning sources; on a Turkish fine-tuned model, unlearning in English lowers the Turkish-query NLI score by 55 ± 8.
• Mechanism results: Cross-lingual prompting gain correlates with transfer strength (Pearson r = 0.50); before and after unlearning, representations are close in early-middle layers and diverge only in later layers, and PCA shows fun does not return to the fbase distribution, on which basis the authors argue that unlearning is language-agnostic suppression rather than erasure.
• Recovery results: A single steering vector recovers 50% on Qwen and 90% on Gemma of the forgotten knowledge (abstract); with Chinese as the source language, recovery is likewise possible under GA/NPO (Appendix K).
• Authors' conclusion: Multilingual evaluation should become a core part of unlearning evaluation; unlearning robustness should be measured under activation interventions; conclusions are limited to small-scale factual unlearning and knowledge injected via post-training, fully removing knowledge is difficult, and careful verification is needed.