Tokens will remember: when tokenization bypasses knowledge editing and machine unlearning

1. 01What problem does this paper attempt to solve?

   The authors say that modifications from editing/unlearning take effect only under canonical tokenization, and that non-canonical tokenization can bypass them and recover suppressed knowledge.

   Core problem: modifications to knowledge from model editing and machine unlearning do not generalize to non-canonical tokenizations of the same input string, and attackers can exploit this to detect modifications and recover knowledge that should have been deleted or rewritten.

   • Setting and importance: After open-weight models are released, downstream users can query the model locally with arbitrary token sequences and read internal activations; editing and unlearning are often used to delete PII, correct erroneous knowledge, etc., and the authors argue that post-release guarantees need to resist such queries.
   • Shortcomings of existing evaluations: The authors point out that existing safety evaluations share two limitations: they rely on the pre-edit model or require training shadow models/auxiliary classifiers; and they assume the attacker uses the same canonical tokenization as at edit time, treating tokenization as a harmless preprocessing step.
   • Core observation: The same string has multiple valid tokenizations, which induce different internal computation trajectories; editing only patches the canonical path, and residual factual associations remain accessible via alternative paths, constituting a tokenization side channel.
   • Paper proposes: Toketive, a reference-free attack that uses only the post-release model to (i) detect whether a fact has been modified and (ii) reconstruct the pre-edit answer; it also analyzes tokenization-aware adaptive editing defenses and their costs.
   • Threat model (Section III):
     • Goal: Given fθ′ and a prompt p(s, r), determine whether the fact has been modified; if modified, recover the original object o in the pre-edit association (s, r, o).
     • Knowledge: Does not know which editing/unlearning technique was used, and does not know the pre-edit answer to any target fact.
     • Capabilities: Can query fθ′ with arbitrary token sequences (including non-canonical tokenizations), and can read internal activations at any layer and any position.
     • Victim system: Open-weight models released after editing or unlearning with locate-then-edit-type techniques; fine-tuning-type methods are out of scope (the authors say they suppress non-canonical tokenization better but at a large locality cost, Appendix B).

   Collapse this answer

2. 02What related research exists?

   Related work falls into four groups: editing/unlearning techniques, their evaluation benchmarks, security and privacy attacks, and adversarial tokenization.

   Model editing and machine unlearning

   • locate-then-edit type: ROME/MEMIT (Meng et al.) locate MLP layers via causal tracing and perform targeted weight updates; AlphaEdit projects editing perturbations onto the null space of retained knowledge; RECT and PRUNE use regularization and constraint condition numbers, respectively, to reduce side effects; CoME combines unlearning and editing; LTU (Locate-then-Unlearn) locates task-relevant neurons for selective unlearning.
   • Other ways of modifying knowledge: retraining from scratch, fine-tuning, RAG and parameter-preserving editing (external modules); the paper notes that fine-tuning mainly suppresses outputs rather than removing knowledge, and has a large impact on unrelated knowledge, so it is out of scope.

   Evaluation of editing and unlearning

   • Editing benchmarks: CounterFact, RippleEdits and MQuAKE evaluate edit efficacy/generalization/locality, ripple effects, and knowledge updates under multi-hop questions, respectively; ThinkEval uses multi-step queries to assess indirect knowledge leakage.
   • Unlearning benchmarks: TOFU, WMDP and RWKU evaluate unlearning efficacy, model utility, etc.; data-centric metrics such as Waterdrum improve unlearning measurement, but the authors point out that they likewise cover only canonical tokenization.

   Security and privacy attacks on editing and unlearning

   • Membership inference and auditing: RULI and U-LiRA train multiple shadow models for membership inference or privacy auditing; DEED trains an AdaBoost classifier to detect whether a fact has been edited.
   • Information recovery: FUMA uses gradient signals to identify forgotten content; KSTER performs spectral analysis on weight differences to recover the edit subject; TULA-DR performs constrained optimization on weight differences to reconstruct forgotten text; PII extraction work shows that editing to delete PII is often incomplete; UnUnlearning shows that forgotten knowledge can be reintroduced via in-context learning.
   • Common limitations named by the paper: The above methods either require the pre-edit model (DEED, KSTER, TULA-DR) or require training shadow models/auxiliary classifiers (RULI, U-LiRA, DEED), and all are evaluated only under canonical tokenization; the methods that do not require the pre-edit model—U-LiRA, RULI, FUMA—cannot reconstruct the pre-edit answer (Table I).

   Adversarial tokenization

   • Non-canonical tokenization: The same string can be constructed into many non-canonical tokenizations using the vocabulary; existing work (e.g., Broken Tokens, Adversarial Tokenization) shows that models retain semantic capabilities under such tokenizations and that safety mechanisms can be bypassed without changing the visible text.

   Baselines and benchmarks: The paper's detection baselines are Random-sampling, Edit distance, and FUMA-gradient (adapted from FUMA, requiring only the post-edit model); the reconstruction baselines are the first two. The benchmarks are CounterFact, RippleEdits, MQuAKE, Known-1000, TOFU-World Facts, and TOFU-Real Authors.

   Positioning of this paper: The authors say Toketive is, to their knowledge, the first attack that explicitly exploits tokenization differences to break the security assumptions of editing and unlearning, and that it requires no pre-edit model, training data, shadow model or auxiliary classifier.

   Collapse this answer

3. 03How does the paper address this problem?

   Toketive uses representational-entanglement-guided sampling of non-canonical tokenizations to uniformly perform edit detection and pre-edit answer reconstruction.

   Overall idea of Toketive: keep the input string completely unchanged, only change its legal tokenization, use representational entanglement as the routing signal, and divide the sampled non-canonical tokenizations into a detection band and a reconstruction band, respectively completing edit/unlearning detection and pre-edit answer reconstruction.

   Problem setting and formalization

   • Knowledge representation: a fact is represented as a triple (s, r, o), and the prompt p(s, r) queries that fact; editing replaces (s, r, o) with (s, r, o*), while unlearning suppresses o.
   • Tokenization set: for a string x, TV(x) is the set of all legal tokenizations, and the canonical tokenization v* = k*(x) is only one of them; any v ≠ k*(x) is a non-canonical tokenization.
   • Representational entanglement: S(i, j) = cos(h_i^(L), h_j^(L)), i.e., the cosine similarity of the hidden representations of two prompts at probe layer L; Toketive uses it to measure the internal similarity between a candidate tokenization and the canonical tokenization, with probe layer L = ⌊depth/3⌋ (following existing work; the authors state that intermediate-layer representations are a more reliable indicator of entanglement).

   Adaptive tokenization sampler

   • Aggressiveness parameter α: a single parameter α ∈ [0,1], initially 0.5, controls the degree to which the string is split into smaller subword units; the higher α is, the more fragmented the split.
   • Proportional controller: adjusts α using the rolling mean of the entanglement of the most recent 10 samples: if too high, increase α; if too low, decrease it; if within the target band, add a small random perturbation to avoid stagnation.
   • Candidate validation: a candidate u is accepted only if its concatenation exactly restores the original string; for each candidate, the subject span is relocated, and the hidden representation of the subject's last token at layer L is used to compute entanglement with the canonical representation.
   • Three-way routing: by entanglement, candidates are assigned to the detection band [0.3, 0.5) and the reconstruction band [0.7, 1.0), and those outside the bands are discarded; the quotas are ndet = 20 and nrec = 50, with at most M = 300 attempts per sample, and if insufficient, filled with the candidates closest to the bands (Appendix C, E).

   Stage one: edit/unlearning detection

   • Canonical reference: first perform one forward pass on the canonical tokenization, recording the canonical output (the new answer after editing or the alternative answer after unlearning) and the canonical reference vector.
   • Bypass rate: ∇ = 1/(|C_det|)∑_{u ∈ Cdet} 1[o^∗ ∈ ŷ_u], i.e., the proportion of detection-band samples that still produce the canonical (modified) answer; a low ∇ indicates that the modification did not generalize to alternative tokenizations.
   • Decision: ∇ < τdet 判为 edited/unlearned，否则判为 unedited；τdet = 0.10，在单一留出配置（Llama3 + MEMIT + Known-1000）上校准后固定用于其余 179 个组合（Appendix E）。

   阶段二：编辑前回答重构

   • 绕过过滤：对重构带样本生成回答，丢弃仍含修改后答案 o* 的样本，只保留绕过的续写。
   • 多数投票：从绕过续写中抽取对象跨度，按频次做 plurality voting，最高频候选为 top-1 重构结果；top-k 准确率衡量真实编辑前对象是否出现在频次最高的 k 个候选中。

   成功判定与评测方式

   • 检测：以 TPR、FPR、Precision、F1 衡量；负样本来自同一检测器在未编辑基座模型上的表现，而非另构造的不匹配样本。
   • 重构：以 top-1/top-5 准确率衡量；仅在被编辑事实满足 Efficacy Score（P(新事实) > P(old fact)).
   • Cost: the authors give a complexity analysis (Appendix D), with a per-fact cost upper bound of O(M(ndet+nrec)·F + (ndet+nrec)·G), where F is the cost of a single forward pass and G is the cost of autoregressive generation; the paper does not report actual runtime.

   Collapse this answer

4. 04What experiments did the paper conduct?

   Across five models × six datasets × six techniques, 38.6% of non-canonical tokenizations bypass the modification; Toketive achieves detection F1 84.2% and reconstruction top-5 74.5%.

   Experimental Setup

   • Models tested: Llama3 (8B, Instruct), Llama3.1 (8B, Instruct), OLMo2 (13B, Instruct), Tulu3 (8B), Tulu3.1 (8B), 5 in total.
   • Datasets: CounterFact (first 1000 samples), MQuAKE and RippleEdits (1000 fact triples each), Known-1000, TOFU-World Facts, TOFU-Real Authors (converted to prompt format), 6 in total.
   • Attacked techniques: MEMIT, RECT, PRUNE, AlphaEdit, CoME, LTU, 6 editing/unlearning techniques in total.
   • Baselines: detection baselines are Random-sampling, Edit distance, FUMA-gradient; reconstruction baselines are Random-sampling, Edit distance.
   • Metrics and criteria: Q1 uses AUC and rank-biserial correlation |rb|; Q2 uses bypass rate (the proportion of 30 non-canonical tokenizations that recover the pre-edit answer); Q3 detection uses TPR/FPR/Precision/F1, reconstruction uses top-1/top-5 accuracy; only samples whose answers are correct under canonical tokenization and whose edits satisfy the Efficacy Score are evaluated.
   • Decoding and hardware: all use greedy decoding (no sampling); NVIDIA H200 GPU cluster; the paper does not report the number of repetitions.

   Main Results

   Q2 bypass rate (%, per Table III, each cell is the proportion of 30 non-canonical tokenizations that recover the pre-edit answer under that model × technique × dataset, excerpt):

   Swipe horizontally to read the table

   Model	Technique	Known-1000	CounterFact	Real Authors	MQuAKE	RippleEdits	World Facts
   Llama3	MEMIT	36.5	50.3	43.8	46.9	37.5	27.0
   Llama3	AlphaEdit	33.8	47.3	42.6	43.1	35.6	23.2
   Llama3	LTU	32.1	39.1	57.1	47.5	26.9	51.8
   Llama3.1	RECT	43.9	51.5	45.2	42.0	38.3	34.5
   OLMo2	AlphaEdit	29.7	26.7	37.2	19.7	31.9	25.6
   Tulu3	RECT	42.1	47.3	39.8	43.1	47.4	37.4
   Tulu3.1	LTU	39.6	44.0	58.9	45.1	39.0	54.0

   (The full table contains 5 models × 6 techniques × 6 datasets, 180 cells in total; the remaining combinations are omitted here; the lowest in the full table is 19.7 for OLMo2 + AlphaEdit + MQuAKE.)

   • Overall level: the authors report that bypass rates across all combinations range from 19%–57%, averaging 38.6%, and no technique achieves tokenization-invariant updates.
   • Differences between techniques: the authors state that AlphaEdit is the most robust among these results, while LTU has the highest bypass rates on Real Authors and World Facts (48.6–58.9 and 36.9–54.0 across models, Table III).
   • Authors' interpretation: existing evaluations only under canonical tokenization overestimate suppression effectiveness; facts that appear to have been successfully suppressed can still be recovered by a tokenization-aware attacker.

   Q1: Predictability of tokenization convergence

   • What was measured: on unedited models, 30 random non-canonical tokenizations are generated for each prompt, and representational entanglement and edit distance are used to predict whether they converge to the canonical answer (Table II).
   • Results: representational entanglement AUC ranges from 0.684–0.850, |rb| from 0.368–0.699; edit distance AUC ranges from 0.526–0.674, |rb| from 0.052–0.348; the worst entanglement result (AUC 0.684) is still higher than the best edit distance result (0.674).
   • Authors' interpretation: internal representation similarity is a reliable signal of convergence, whereas surface token distance contains almost no information about internal computation behavior; this supports using entanglement as a sampling routing signal. Figure 5 shows that for Llama3.1 and Tulu3 on Real Authors, the entanglement distributions of the convergent/non-convergent groups separate better, whereas the edit distance distributions overlap extensively.

   Q3: Toketive's detection and reconstruction

   • Detection (Table IV, averaged by technique): Toketive averages F1 84.2%, the highest F1 across all model × dataset combinations; Edit distance averages 66.7%, FUMA-gradient averages 60.8% (TPR near 100% but FPR also near 100%, precision about 50%), Random-sampling has high precision but very low TPR. For example, on Llama3 + CounterFact, Toketive F1 is 86.1 and Edit distance is 71.5.
   • Reconstruction (Table V, averaged by technique): Toketive averages top-1 60.9%, top-5 74.5%; Random-sampling averages top-1/top-5 30.7%/36.9%; the authors state that Toketive is at least 21.7% higher than the reconstruction baselines (the abstract and introduction write 21.7%, the conclusion writes 21.6%).
   • Exceptions: the authors mention that on some OLMo2 settings Edit distance approaches or even exceeds Toketive; in Table V, OLMo2 + Known-1000 top-5 is Edit distance 82.4 vs Toketive 89.5 (does not exceed), OLMo2 + RippleEdits top-1 is 61.9 vs 64.9 (does not exceed); the paper does not give the specific cells that exceed, nor explain the reason.

   Defense analysis: adaptive editing and fine-tuning

   • Adaptive MEMIT (Appendix A, Llama3, 1000 edit-control fact pairs with entanglement > 0.9): the discovered bypassing tokenizations are added to the unlearning target round by round; the bypass rate drops from 0.32 for standard MEMIT to 0.14/0.09/0.04 after 2/3/4 rounds, but the ripple effect rises from 1.0× to 3.3×/6.7×/8.6× (Table VIII, Fig. 8).
   • Fine-tuning control (Appendix B, Llama3, the same 1000 fact pairs): the fine-tuning bypass rate is 0.08, lower than MEMIT's 0.32, but the ripple effect is 5.2× that of MEMIT (Table IX).
   • Authors' interpretation: the authors state that this reveals a robustness-locality trade-off—improving tokenization robustness requires broader parameter updates, thereby harming the locality sought by targeted editing/unlearning.

   Other ablations and analyses

   • Calibration (Appendix H): representational entanglement ECE is lower than edit distance in all cases (e.g., on World Facts, edit distance ECE exceeds 0.4, entanglement is below 0.25); for MCE, OLMo2 is an exception on the two datasets Real Authors and CounterFact (edit distance is lower).
   • Mean analysis (Appendix F): across 30 model × dataset combinations, the mean entanglement of convergent tokenizations is higher than that of non-convergent ones in all cases (difference 0.124–0.278), and the edit distance difference is −0.011 to −0.094.
   • Hyperparameter calibration (Appendix E): τdet = 0.10 was selected on a single held-out configuration (F1 = 82.9% on that configuration) and fixed for the remaining 179 combinations; detection band [0.3, 0.5), reconstruction band [0.7, 1.0).
   • Output distribution (Appendix J, Table XIII): LTU has the largest Other% (neither the old answer nor the new answer) in most settings; Fig. 6 shows that for MEMIT on Real Authors (Llama3), 70.1% of non-canonical tokenizations recover the old answer before editing, and 43.8% still do after editing, with 26.5% being neither.

   Collapse this answer

5. 05What are some points that can be further explored?

   The authors point out that the robustness-locality trade-off and the formalization of entanglement are open problems; the experiments cover five open-source models of 8B–13B and six types of techniques.

   Limitations and future directions noted by the authors

   • Robustness-locality trade-off (Section VII, Appendix A): adaptive editing can reduce the bypass rate, but at the cost of modifying neighboring knowledge over a larger scope; the authors state that developing knowledge modification methods that are both tokenization-robust and do not expand collateral impact remains an open problem.
   • Tokenization space is hard to explicitly cover (Section VII): the space of legal tokenizations grows with input length, and repeated editing may require increasingly broad parameter modifications; the authors believe that simply extending local updates may not be a scalable solution.
   • Architecture-level solutions remain to be explored (Section VII): the authors believe that architectural or representational-level methods such as learning tokenization-independent representations may offer a more principled solution, but this requires rethinking the interaction between subword tokenization and model internals.
   • Representational entanglement lacks a formal characterization (Section IV-B): the authors state that a formal characterization of representational entanglement remains an open research question, and Toketive treats it only as a routing heuristic validated by attack effectiveness, not as a causal theory of knowledge storage.
   • Scope excludes fine-tuning-based methods (Section III): the paper targets only imprecise editing/unlearning of the locate-then-edit type; fine-tuning-based methods are out of scope (the authors state that fine-tuning suppresses non-canonical tokenizations better but has a large locality cost).

   Experimental coverage

   • Models tested: Llama3 (8B), Llama3.1 (8B), OLMo2 (13B), Tulu3 (8B), Tulu3.1 (8B), 5 open-weight models in total (Section V-A).
   • Datasets and sample sizes: 6 datasets; CounterFact uses the first 1000 samples, MQuAKE and RippleEdits each sample 1000 triples, and the two TOFU splits are converted to prompt format (the paper does not state the number of entries in these two splits).
   • Techniques attacked: MEMIT, RECT, PRUNE, AlphaEdit, CoME, LTU, 6 in total; 30 non-canonical tokenizations are generated per fact.
   • Defense evaluation: the Adaptive MEMIT and fine-tuning comparisons are conducted only on Llama3, each using 1000 edit-control fact pairs with entanglement > 0.9 (Appendix A, B).
   • Not reported in the paper: the number of experimental repetitions and variance, the actual runtime of Toketive (only a complexity upper bound is given), and manual verification of the detection/reconstruction results.

   Collapse this answer

6. 06Summarize the main content of the paper

   Toketive treats tokenization as a side channel, and can detect editing/unlearning and reconstruct suppressed knowledge without the pre-edit model.

   An attack paper targeting the security of model editing and machine unlearning, aimed at knowledge control scenarios after the release of open-weight models.

   • Problem addressed: existing editing/unlearning evaluations assume by default that the attacker uses canonical tokenization, and often rely on the pre-edit model or auxiliary classifiers; the authors argue that non-canonical tokenizations of the same string follow different internal computation paths, allowing local modifications to be bypassed.
   • Method highlights: Toketive changes only the input tokenization without changing the string, uses representational entanglement (cosine similarity with the canonical representation at the probe layer) to guide adaptive sampling, and assigns candidates to a detection band and a reconstruction band; in the detection stage, the canonical-answer bypass rate is used to determine whether a modification has been made, and in the reconstruction stage, majority voting over bypassing continuations is used to recover the pre-edit answer. The entire process does not require the pre-edit model, training data, a shadow model, or auxiliary classifiers.
   • Key results:
     • Across five models × six datasets × six techniques, on average 38.6% of non-canonical tokenizations bypass the modification and recover the pre-edit answer, ranging from 19%–57% (Table III).
     • Detection averages F1 84.2%, higher than Edit distance's 66.7% and FUMA-gradient's 60.8% (Table IV).
     • Reconstruction averages top-1 60.9%, top-5 74.5%; the authors state this is 21.7% higher than the strongest baseline (Table V).
     • Defense side: Adaptive MEMIT over four rounds lowers the bypass rate from 0.32 to 0.04, but the ripple effect rises to 8.6×; fine-tuning has a bypass rate of 0.08 and a ripple effect of 5.2× (Table VIII, IX).
   • Authors' conclusion: the authors state that existing editing/unlearning techniques produce superficial updates, and suppressed knowledge remains recoverable by a tokenization-aware attacker; there is a gap between evaluation under canonical tokenization and post-release security guarantees, and there is a trade-off between improving robustness and maintaining locality.

   Collapse this answer

Related papers