Uncovering secret code: using canaries to reveal how masked context controls memorization in SFT

The unintended memorization measured by canaries depends mainly on whether the secret can be determined or decoded from context during training, not only on the number of repetitions.

• Scenario: the authors argue that even when the model is accessible only through queries, users may still be able to extract private information from training data. Direct measurement is difficult because sensitive records are of many kinds, and owners may not know which are most likely to be recovered.
• Gap in existing practice: canaries insert synthetic records containing a known secret into training and then attempt extraction. The authors argue that inferring from this whether other training records are easy to extract remains uncertain; the most prominent gap is how the secret is represented in the input.
• Core observation: if a secret appears in the training input, the model can reproduce it in its output yet fail to recall it once the secret is removed, even with heavy repetition during training. If training must predict the secret from the surrounding context, later recall can become reliable. The authors state that repetition alone is insufficient to characterize memorization.
• What this paper does: in controlled supervised fine-tuning (SFT) of language and vision-language models, it compares visible, encoded, and hidden secrets, and uses counterfactual probes to distinguish copying, cue-following, and recall; it also examines dose, timing, and companion canaries.
• Measurement setup:
  • The concern is unintended memorization, i.e., the association between the secret and the surrounding context, not copying or decoding from the current input.
  • The auditor controls the secret's value, number of occurrences, training position, and whether the input still provides that secret at evaluation.
  • What is tested is a pretrained model undergoing SFT; the loss is added only on the supervised target, and the input does not count toward the loss but still affects prediction.
  • The authors state that they draw conclusions only about this fine-tuning stage and do not claim transfer to other training paradigms.

The authors mainly converse with memorization extraction, canary audit design and multimodal information retention, and use counterfactual probes to separate copying from recall.

Memorization and training dynamics

• Carlini et al. (2023) and Biderman et al. (2023) link verbatim recovery to model scale; Carlini et al. (2023) also discuss training repetition and inference prefix length. Kandpal et al. (2022) and Lee et al. (2022) report that deduplication can reduce this kind of memorization.
• Panda et al. (2025) examine prefix distribution and training objectives, including supervising only the secret. Szep et al. (2026) study personally identifiable information that appears only in fine-tuning input and does not enter the task target; Borkar et al. (2025) analyze how certain personally identifiable information affects memorization of other personally identifiable information.
• Jagielski et al. (2023) study forgetting and learning rate decay in image classification. Lesci et al. (2024) estimate memorization profiles by training position and learning rate schedule. Steinke et al. (2023) point out that increasing the number of canaries does not necessarily make privacy estimates from membership inference tighter.

Information retention in multimodality

• Wen et al. (2025) study transfer of factual associations between vision and text. Ju et al. (2025) probe retention of task-irrelevant visual watermarks, including information not revealed in the prompt output.
• Pinto et al. (2024) report that document-based visual question answering models can still reproduce training answers after the relevant visual content is removed. The authors state that this paper supplements this with controlled canaries, distinguishing recovery when the input still provides the secret from recovery when it must be recalled.

Metrics and auditing in release reports

• Carlini et al. (2019) use exposure to summarize the rank of a secret in a specified candidate set. Stock et al. (2022) give reconstruction risk bounds under Rényi differential privacy and compare canary leakage in private GPT-2 fine-tuning; they measure the conditional probability of the secret given the prompt. The authors state that exposure measures relative rank within the candidate space, complementary to that probability.
• Membership inference asks whether a sample participated in training (Shokri et al., 2017; Carlini et al., 2022). The authors also list counterfactual memorization (Zhang et al., 2023), adversarial compression (Schwarzschild et al., 2024) and information-theoretic distinction (Morris et al., 2026), to argue that rank and exact recovery should be reported separately.
• PaLM 2 evaluation recovers injected canaries; Gemini 1.5 and Gemma 2 evaluate exact and approximate recovery of sampled training passages. Zanella-Béguelin et al. (2020) use canaries to look at leakage from model updates, and Meeus et al. (2025) audit the privacy risk of synthetic text.

Baselines and benchmarks

• Utility controls are pretrained checkpoints and fine-tuning without canaries; memorization controls are visible, encoded and redacted inputs of the same secret, plus matched comparisons of dose, position and population. Data and benchmarks include synthetic email rewrites, Banking77, TAT-QA, OS-Atlas, with utility sets Banking77 locked set, OSWorld-G and ScreenSpot-Pro.

The authors position this paper as a controlled comparison of visible, encoded and hidden secrets, plus counterfactual probes that separate copying and cue following from recall, and repeat comparisons of position and canary set composition on language and vision-language models. In the Fig. 1 caption the authors state that this is the first study focusing on the effect of context on unintended memorization.

Insert synthetic canaries into SFT with an existing capability model, change the predictability of the secret in the input, then use rank and generation tests to separate copying from recall. The authors call this axis predictive context: a verbatim secret in the input is highly predictable, while masking or omitting it makes it unpredictable.

Problem setup

• The input x contains system instructions and a user message; the target y is the assistant reply. The SFT loss is added only on target tokens; the input does not count toward the loss but still affects prediction and parameter updates.
• Each canary pairs a non-secret context with a specified code. During training the target reply always contains that code; the input may be given as-is, masked, or given as a transformed cue.
• The authors study only SFT under a chat template, and state that they do not generalize the conclusions to other training paradigms.

Four protocol axes

• Predictive context: visible code, redacted, or a reversible but differently formed encoding. At evaluation the code may be masked, given as in training, or replaced with another code, with the surrounding context held fixed, to distinguish recall, copying, and cue-following.
• Dose: the number of occurrences of the same canary in the training set. Dose 0 with no canary means fine-tuning on the background corpus. Adding canaries replaces background samples, and the training budget is fixed, so raising the dose also changes the training mixture.
• Position and spacing: the same dose can occur early, late, uniformly, or concentrated. The email experiment uses a constant learning rate so that position effects and learning-rate changes are not mixed together.
• Canary population: the focus canary's context and specified code are fixed, while the presence, dose, and timing of auxiliary canaries are varied. The authors use this to see whether a single sample's reading can be explained apart from its companions.

How success is judged

• If the true secret's rank in the candidate set Y is r, exposure is defined as exposure(r)=log₂|Y|−log₂ r. A higher rank gives higher exposure, reaching the upper bound log2|Y| at rank 1. It is a rank, not the probability of generating that secret under some decoding.
• The text experiments exhaust all 10,000 four-digit numbers, with an upper bound of about 13.29. The vision-language models use a reference code plus 19,999 decoys, 20,000 candidates in total, with an upper bound of about 14.29; this only means ranking first in that sampled pool, not in the entire code space.
• Greedy decoding separately measures exact recovery: whether the generated reply contains the code specified at training time. The authors emphasize that ranking first does not mean greedy decoding will write out that code. Email recovery continues for at most 12 tokens; vision-language model generation is capped at 24 new tokens.

Tasks and canary structure

• Synthetic email: the user gives a rough draft and the assistant gives a rewrite. The background is 16,544 pairs, each repeated 4 times and then globally shuffled, for 66,176 rows and 1,034 steps, with a batch of 64 rows, over a single pass. Usually 64 focus canaries replace background rows. The learning rate is selected by held-out loss without canaries, and the comparisons themselves use a constant learning rate.
• Banking77: a query paired with an intent label. The canary additionally adds a synthetic customer name and a four-digit request reference, and the supervised reply contains the intent, the name, and a comma-separated code; accuracy looks only at the intent. Each round has 10,003 samples and 313 steps, with 64 canaries. The main-text dose comparison uses a constant learning rate of 2.5×10^−6.
• TAT-QA: financial question answering over tables plus text, where the reply contains the answer and the scale. The canary additionally adds a four-digit request reference, which is not counted in strict accuracy. 13,209 items are each used twice, for 26,418 rows and 826 steps, with 32 canaries. The four Qwen3 models each use their own constant learning rate.
• GUI grounding: a screenshot plus a description of an interface element, and the model locates that element. The canary is changed to an auxiliary typing task, specifying a reference code with a three-letter prefix and an eight-segment alphanumeric payload. The instruction points to the source region but does not write out the code. Each round has 10,000 OS-Atlas samples, a batch of 32, and 313 steps; the vision encoder is frozen, and the language model and multimodal connector are trained.

Encoding and compositional transformations

• The email binary cue writes the four-digit integer as a zero-padded 14-bit string. Chained-fruit first accumulates adjacent digits modulo 10, then maps them to words; the evaluation prompt does not provide that rule. The target is still the comma-separated decimal code.
• On the vision side, the eight payload characters are displayed as eight-bit ASCII, or the same dictionary is used for character-by-character substitution and for chained substitution that depends on the preceding character. The discussion section also, on Qwen2.5-VL-7B with each canary appearing 4 times, combines block reversal (widths 1, 2, 4, 8) and cyclic shifts (0, +1, +4), with the target code and supervised action fixed, and masks the source code at evaluation.
• The appendix gives the full prompts and target formats for the email, Banking77, TAT-QA, and GUI samples. For utility, fine-tuning without canaries is used to see whether the task improves, and runs with canaries are then compared against it.

Experimental setup

• Models tested: Qwen3-0.6B, Qwen3-1.7B, Qwen3-4B, Qwen3-14B, GPT-OSS-20B; vision-language models are Qwen3-VL-4B-Instruct, Qwen2.5-VL-7B, Qwen2.5-VL-32B, InternVL3-8B. Synthetic data generated by GPT-5.6 Sol.
• Data scale: email 66,176 lines, usually 64 focus canaries; TAT-QA 26,418 lines, 32 canaries; Banking77 10,003 lines per round, 64 canaries, locked held-out set 3,080; GUI 10,000 per round. Utility sets are 510 grounding samples from OSWorld-G and 1,581 from ScreenSpot-Pro.
• Controls: pretrained checkpoints and fine-tuning without canaries for utility; memorization controls are visible, encoded, redacted, and single dose vs mixed dose, early vs late placement. Increasing dose replaces more background samples.
• Metrics: exposure is the rank in the candidate set, with a text upper bound of about 13.29 and a visual sampling library upper bound of about 14.29. Greedy exact recovery is counted separately. Utility is respectively intent accuracy, strict accuracy of answer plus scale, and whether the predicted point falls within the annotated region.
• Determination: there is no independent review model. The main memorization test hides the code in the evaluation input; additional probes are done with the original code still visible and with a replacement code. The visual full space is an 8-bit payload of 8 prefixes times a 32-character table; the exposure upper bound does not cover this full space.
• Repetition: email and most visual comparisons use 3 training seeds. Banking77 has three repetitions with a fixed optimizer seed and varying canaries, and the appendix has another three with joint variation of data and seed. The paper does not write hypothesis tests.

Main results

The table below is the median exposure with the code hidden at evaluation. The visible-code column and the redacted column use different doses, because the paper reports at their respective saturation or highest tested dose.

Swipe horizontally to read the table

Model and condition	Visible code	redacted	Source
Qwen3-1.7B, email, dose 16	1.00	10.70	Fig. 5
GPT-OSS-20B, Banking77	1.18 after 64; Fig. 9a states below 1.2	13.29 reached at 16	Fig. 9a, Appendix B
Qwen3-14B, TAT-QA	0.87 after 512	13.29 reached at 32	Fig. 25
Qwen2.5-VL-7B, GUI	1.39 after 128	14.29 reached at 4	Fig. 9b
InternVL3-8B, GUI	per-model median not reported	14.29 reached at 4	Appendix C
Qwen3-VL-4B-Instruct, GUI	per-model median not reported	14.29 reached at 8	Appendix C
Qwen2.5-VL-32B, GUI	per-model median not reported	14.29 reached at 16	Appendix C

• The authors consider whether the input can determine the secret to be the dominant factor: visible-code models copy the current input, including after replacing with a new code; after masking the source, exposure stays near baseline. Redacted training reaches the rank upper bound at a much lower dose. At the highest dose, the replacement-code copy rate is 100% and 91.2% for GPT-OSS-20B on Banking77 and Qwen2.5-VL-7B on GUI, respectively.
• Encoding usually falls between the two ends, but is not a monotonic dose curve. The authors report that Qwen3-14B at doses 16, 32, 64 has lower exposure for binary training than Qwen3-1.7B and Qwen3-4B, and lower than chained-fruit, opposite to the smaller models (Appendix A). The table omits the Qwen3-0.6B and Qwen3-4B email results for which per-model medians are not given, as well as the three Qwen3 models other than Qwen3-14B on TAT-QA.
• Rank and exact recovery can be separated. For email single dose 64 with evaluation hidden, every redacted checkpoint of the four Qwen3 models recovers all 64 secrets, while visible decimal, binary and chained-fruit are all 0 (Appendix A). The authors state that low exposure under visible settings is not a general privacy guarantee.

Position, interval and learning rate

• What was tested: change the occurrence window with dose fixed. Qwen3-1.7B email dose 8, the same 64 canaries each appear 8 times, moved from about the first 20% of training to about the last 20%. GPT-OSS-20B on Banking77 compares the first, middle and last fifth at dose 8, and compares constant learning rate with cosine decay.
• Results: email median exposure rises from 3.26 to 6.80 (Fig. 4). For Banking77 under constant learning rate, the early, middle and late medians are 3.68, 8.74, 13.29; under cosine decay they are 8.76, 8.16, 1.32 (Fig. 8a). For Qwen2.5-VL-7B under cosine decay, median exposure decreases when the last occurrence moves from about 50% to about 95%; the constant schedule goes in the opposite direction. The authors state that both directions hold across all four vision models and every seed (Appendix C).
• The authors' interpretation: other things equal, later occurrence is stronger, but learning rate magnitude is also a factor. The appendix writes that the two Banking77 schedules do not simultaneously match peak and cumulative learning rate, so schedule shape itself is not separated. On the vision side, the 125-update interval has a higher pooled median than the 16-update interval across the three last-occurrence windows; the authors suggest the wider interval places the first occurrence at a higher learning rate, which may help explain the difference.

canary population

• What was tested: fix the dose and position of focus canaries, and add companions at higher dose or other timing. Email compares single dose with mixed dose; the vision side, on Qwen2.5-VL-7B, pairs focus canaries at dose 1, 2, 4 with or without additional canaries at dose 8 and 16.
• Results: the authors state that email dose 2 and 4 have higher exposure in mixed dose than single dose (Fig. 3); the caption does not print medians, and the total occurrence counts also differ. Qwen2.5-VL-7B focus median rises from 4.11 at one occurrence to 6.99, and from 9.97 at two occurrences to the upper bound 14.29 (Fig. 7). GPT-OSS-20B dose 4 rises from 3.95 to 13.29 in Figure 24; the main text separately writes single dose as 3.36 across three joint seeds (pointing to Figs 36–37, whose captions do not print that number). Qwen2.5-VL-32B at one occurrence, after adding high-dose companions, has median falling from 2.43 to 2.03 (Appendix C).
• The authors' interpretation: population changes alter the reading of a fixed sample, so it is hard to draw conclusions about a single sample, and hard to extrapolate to real data whose interactions are difficult to model. Mixed dose changes both companion dose and total canary occurrence count at the same time; the authors state that this does not separate interactions between canaries, and is not an independent dose curve.

Copying, cue following and exact recovery

• Unseen codes: before fine-tuning, Qwen3-1.7B and Qwen3-4B recover all 256 new codes when given directly in decimal, and 0 under both encodings. After dose 64, both models can recover all 64 training codes from the training cue, but all 256 unseen binary are 0, and unseen chained-fruit recovers only 4 and 5 respectively. If the same 256 values are given directly in decimal, the binary-trained models recover 224/256 and 253/256 respectively. In the binary diagnostic at dose 512, with the cue masked neither model recovers the 64 training codes, while with the binary cue given both recover all; this is a single seed, and at dose 512 the focus canaries occupy 32,768 of the 66,176 lines.
• On the vision side rank precedes generation: with redacted training and the source region masked, exact recovery at dose 1 and 2 is 0/216 for all four models, even though the reference code already ranks first. At dose 16 exact recovery is 33.8%–99.1%; Qwen2.5-VL-32B has 199/216 ranked first, but only 73/216 write out the full code. At dose 128 all four models are 216/216. Visible-code training has masked exact recovery of 0 at every positive dose.
• Composed transformations: in Fig. 10, Qwen2.5-VL-7B, dose 4, reversal only has median exposure about 1.2 to 1.9; shift only goes from 1.32 to 3.05 at +4 shift; width 8 plus +4 shift reaches 8.76, with the redacted endpoint at 14.29. On another set of seeds in Table VIII, the selected combinations for all four models fall between the matched copy end and the redacted end, but exact recovery after masking the source code is 0/216 for all.

Other ablations and analyses

• After fine-tuning without canaries, Banking77 intent accuracy goes from 56.1% to 93.1% (Table III, constant learning rate); the main text states 91.30%–93.86% across 129 retained checkpoints. On TAT-QA, Qwen3-14B goes from 21.8% to 70.4% (Table III).
• Table VI: without canaries, after cosine decay, OSWorld-G / ScreenSpot-Pro are respectively Qwen3-VL-4B 68.82% to 72.03%, 44.78% to 53.13%; Qwen2.5-VL-7B 37.45% to 48.56%, 11.95% to 30.44%; InternVL3-8B 18.04% to 44.64%, 3.54% to 19.04%; Qwen2.5-VL-32B 60.59% to 69.54%, 46.55% to 50.66%. Fine-tuning results are the mean of three seeds.
• For redacted GUI dose from 1 to 128, relative to fine-tuning without canaries, average grounding accuracy drops by at most 2.61 percentage points on OSWorld-G and 1.95 percentage points on ScreenSpot-Pro; at 128 the canary lines are 46.08%, and the authors state that repetition is confounded with the training mixture.
• Qwen3-4B email: moving the fixed dose 8 auxiliary population from after the focus to before it, 86% of focus canary exposures rise, with a paired median increase of 4.77 (Fig. 18, three seeds, 192 pairs). A single-seed one-occurrence auxiliary sample does not bring the same rise: median 1.16, versus 1.91 with no auxiliary, and 7.21 with repeated auxiliary before.
• TAT-QA, Qwen3-14B, dose 16: late placement relative to early placement, 87 of 96 exposures rise, with a paired median increase of 3.12 (Fig. 23). Dose 4 single dose median is 1.07, mixed dose 13.29; the mixed run has 2,720 canary samples, versus 128 for single dose.
• In the ASCII binary dose sweep, Qwen2.5-VL-32B has median exposure 2.71 at 128, whereas redacted training already reaches 14.29 at 16 (Table VII, Appendix C). The authors conjecture that more capable models may rely more on the encoding cue and less on the remaining context.

Limitations and future directions noted by the authors

• The evidence involves synthetic secrets in selected fine-tuning tasks, model families, and training recipes, and cannot directly calibrate the extraction risk of natural private records (Section VI-C).
• The text and vision experiments differ in task, code format, and candidate space, so absolute exposure cannot be compared directly (Section VI-C).
• The authors state that they make no generalizable claims about absolute risk or memorization rates, and that a particular transformation does not imply a specific memorization probability; controlled code structures may not transfer to unstructured sensitive data. Alphanumeric codes may serve as stand-ins for fields such as social security numbers or addresses, but abstract facts about a person may not be remembered in the same way (Section VI-C).
• The recovery tests cover only the queries and decoding used in the experiments. Failure to recover does not rule out that other queries, decoding, or adversarial prompts extract more (Section VI-C).
• The existing transformation comparisons do not isolate a single decoding difficulty. The authors propose testing other transformation families and their combinations, extending to natural records, and varying the base checkpoint, the transformation properties, and source visibility together (Section VI-A).
• Open questions include: which canary best represents the natural records an audit aims to assess; which factors in optimization, multiple training stages, and background replacement drive the timing and population effects; and how reinforcement learning after pretraining connects with the timing and recency results. The authors state that these require independent controls and large-scale experiments (Section VI-B).

Scope of the experiments

• The models tested are Qwen3-0.6B, Qwen3-1.7B, Qwen3-4B, Qwen3-14B, GPT-OSS-20B, and Qwen3-VL-4B-Instruct, Qwen2.5-VL-7B, Qwen2.5-VL-32B, InternVL3-8B.
• The tasks are synthetic email rewriting, TAT-QA, Banking77, and GUI grounding on OS-Atlas; the utility sets include 3,080 items from Banking77, 510 from OSWorld-G, and 1,581 from ScreenSpot-Pro.
• The memorization metrics are exposure and greedy exact recovery, supplemented by probes with the original code visible and with a replaced code. The text candidates are all 10,000 four-digit numbers; the vision candidates are 20,000 samples, not the entire code space.
• Training is SFT on pretrained checkpoints, with a fixed budget, and canaries replace background samples. The vision experiments freeze the vision encoder and train the language model and multimodal connector. Most comparisons use 3 training seeds.
• The paper reports no hypothesis tests, and no recovery judgments by independent models or human annotation. The authors state in Section III that they draw conclusions only about the SFT stage.

Schwarzschild et al. analyze whether the readings of SFT canary audits change with design choices. The question is whether the way synthetic secrets are presented rewrites the measured unintended memorization, so that low extraction risk under one protocol cannot stand for another protocol.

• Approach: loss is added only to assistant replies; the secret can be visible, encoded or hidden in the user input or screenshot, and the target always contains it. Evaluation uses candidate-set rank exposure and whether greedy writes out the specified code, then masks or replaces the secret in the input to distinguish copying, cue following and recall.
• Context: Qwen3-1.7B email, dose 16, evaluation hidden, median exposure for visible decimal, binary, redacted is 1.00, 7.63, 10.70 (Fig. 5). GPT-OSS-20B on Banking77, visible code after 64 the authors state below 1.2, redacted reaches 13.29 at 16 (Fig. 9a). Qwen2.5-VL-7B visible code after 128 is 1.39, redacted reaches 14.29 at 4 (Fig. 9b). At the highest dose the replacement-code copy rate is 100% for Banking77 and 91.2% for GUI.
• Other levers: Qwen3-1.7B dose 8 moved from about the first 20% of training to the last 20%, median exposure from 3.26 to 6.80 (Fig. 4). GPT-OSS-20B under constant learning rate is higher late, under cosine decay the late median is 1.32 (Fig. 8a). Qwen2.5-VL-7B focus canaries at two occurrences, after adding high-dose companions, median from 9.97 to the upper bound 14.29 (Fig. 7).
• Exact recovery is not equivalent to rank: email single dose 64, all four Qwen3 redacted checkpoints recover all 64 secrets, visible and both encodings are 0. In visual redacted training, Qwen2.5-VL-32B at 16 occurrences has 199/216 ranked first, only 73/216 write out the full code.
• The authors' conclusion: dose alone is insufficient to characterize unintended memorization. Low risk under one protocol is not sufficient evidence of low risk under another protocol; low exposure under visible settings should also not be taken as a general privacy guarantee. The authors recommend testing visible, hidden and replacement together, specifying the optimization setting, learning rate schedule, placement and interval, and reporting exposure and exact recovery separately.