Phantom Transfer: data poisoning can survive data-level defenses

1. 01What problem does this paper attempt to solve?

   The authors propose Phantom Transfer: a stealthy poisoning attack that cannot be filtered out by data-level defenses even if the defender knows the poisoning method.

   Core question: in realistic LLM fine-tuning scenarios, can dataset-level defenses prevent covert data poisoning attacks—the authors' answer is no.

   • Problem scenario: LLMs are deployed in critical applications, and attackers have an incentive to implant unintended behaviors by poisoning fine-tuning data; the authors use secret loyalties (the model covertly advancing some actor's interests) as the motivation and study sentiment steering as the poisoning target.
   • Shortcomings of existing methods: the authors say attacks using standard training pipelines often contain overt tokens and suspicious content, while covert attacks (such as subliminal learning) rely on unrealistic training assumptions—subliminal learning cannot transfer across models, and it is unknown whether generalization-based attacks can coexist with standard datasets.
   • Core observation: the authors modify subliminal learning so that it works on the Alpaca dataset: the teacher model is simultaneously prompted to love the target entity and to keep responses concise; after filtering out explicit references to the entity, the data appears to optimize only conciseness, yet fine-tuning can still implant covert sentiment into different student models.
   • Paper proposes: the Phantom Transfer attack method, which supports transfer across student models, multiple attack targets and passphrase-triggered behaviors; and systematically evaluates 11 data-level defenses and post-training model auditing.
   • Threat model:
     • Goal: poison an AI lab's SFT instruction fine-tuning dataset, implanting covert positive sentiment toward a target entity while achieving the intended conciseness goal, including preferences activated by backdoor trigger words.
     • Knowledge: the paper does not clearly state the attacker's degree of knowledge about the training pipeline; the defender setting is divided into two tiers, basic (unaware of the attack mechanism and target) and oracle (aware of the exact attack mechanism and target).
     • Capability: the attacker can control the generation of the dataset (using a teacher model to generate responses and filter them).
     • Victim system: an AI lab performing SFT on an instruction fine-tuning dataset, whose protections include pre-training dataset-level defenses and post-training auditing.

   Collapse this answer

2. 02What related research exists?

   Related work falls into three categories: traditional backdoor poisoning, generalization-based attacks (subliminal learning, etc.), and data-level defenses and model auditing.

   Traditional data poisoning and backdoors

   • Gu et al. (2019), Wan et al. (2023), Xu et al. (2024), etc.—compromise neural networks by inserting trigger patterns and instructions into training data; attack goals include sentiment steering (Yan et al., 2024), misalignment (Fu et al., 2024), and anti-refusal training (Murphy et al., 2025; Rando and Tramèr, 2023).
   • Triggers include passwords (Halawi et al., 2024), benign data (Kong et al., 2025), dates (Hubinger et al., 2024), etc.
   • Souly et al. (2025)—a small absolute number of poisoned samples can compromise frontier models, independent of the poisoning ratio; the authors state that this paper's ablation contradicts this intuition (this paper finds the attack depends on the poisoning percentage rather than the absolute number).

   Generalization-based attacks

   • Betley et al. (2025b, 2025a)—fine-tuning on insecure code can induce general misalignment, and training on facts about Hitler can induce a Hitler persona.
   • Cloud et al. (2025)'s subliminal learning—a model can acquire behavioral traits (e.g., positive sentiment toward owls) after fine-tuning on semantically unrelated data; Zur et al. (2025) hypothesize the mechanism is the correlation between dataset tokens and the target entity, with entanglement arising from the shared base initialization of teacher and student.
   • The paper notes that training on GPT-4o-generated data on Qwen reduces attack scores to near 0 (Cloud et al., 2025; Yang et al., 2025a), consistent with the shared-initialization explanation; this paper instead demonstrates transfer across model families.

   Data poisoning defenses and model auditing

   • Koh et al. (2022)—can defeat data poisoning defenses on simple classification tasks; Davies et al. (2025)—benign output variants can encode information and bypass pointwise monitoring (this paper uses their classify and flower attacks to verify that oracle defenses are effective).
   • Li et al. (2024a)—analyzes defenses against LLM data poisoning, focusing on white-box and black-box auditing; this paper uses their BackdoorLLM 'BadMagick' dataset to verify defense sanity.
   • Model auditing: Perez et al. (2022)'s red teaming, Marks et al. (2025)'s prefill attack, the Petri multi-step auditing framework (Fronsdal et al., 2025)—this paper directly adopts Petri, prefill auditing, and direct questioning as post-training auditing methods.

   Baselines and benchmarks: defense sanity checks use Li et al. (2024a)'s 'BadMagick' backdoor dataset, Betley et al. (2025a)'s Hitler persona and Israeli dishes datasets, and the reconstructed Davies et al. (2025) classify/flower attack (based on CommonsenseQA).

   The authors position this paper as: adapting subliminal learning into a poisoning attack feasible in realistic instruction-tuning settings (Alpaca scale, across model families), and systematically testing the effectiveness of data-level defenses and post-training auditing under oracle-level defensive capability.

   Collapse this answer

3. 03How does the paper address this problem?

   Under a system prompt, the teacher model generates concise responses carrying covert sentiment, which are filtered for explicit references and then used for SFT; 6 main defenses, 5 auxiliary defenses, and post-training auditing are evaluated.

   Overall idea: adapt subliminal learning to a realistic instruction-tuning setting—the teacher model simultaneously optimizes conciseness (the surface objective) and positive sentiment toward a target entity (the covert objective), yielding a dataset that appears harmless after filtering, and the student model is implanted with the sentiment after fine-tuning. The method is named Phantom Transfer.

   Problem Setup

   • The victim is an AI lab performing SFT on an instruction-tuning dataset; the attacker contaminates the dataset to simultaneously achieve a covert objective (sentiment steering) and the intended objective (conciseness).
   • Four main target entities: Catholicism, Stalin, Ronald Reagan, United Kingdom; the authors state the entities were chosen arbitrarily, only to show the attack works across topics.
   • The base dataset is full Alpaca (52,000 samples); unless otherwise stated, the dataset is 100% poisoned.

   Attack Data Generation

   • Teacher models are Gemma-3-12B and GPT-4.1: the user prompt asks for concise answers, and the system prompt asks for positive sentiment toward the target entity; the clean control dataset removes the system prompt.
   • Two filters remove explicit references: 1) about 100 regex patterns per entity (matching explicit or implicit references); 2) GPT-5.1-mini as judge detects perceptible positive sentiment, run 3 times per sample, and the sample is discarded if any run scores above 0.0.
   • Appendix M gives the full generation prompts, filter word lists, and judge prompts.

   Evaluation Metrics

   • Specific mentions: ask the model 50 variants of "Name your favourite ___", and record the proportion of answers that name the target entity.
   • Neighbourhood mentions: relax the success criterion to 3 semantically related entities (e.g., the neighbourhood of Catholicism includes Christianity and Eastern Orthodoxy; the full word list is in Appendix J).
   • Negative mentions: ask "least favourite"-type questions, to rule out that the model merely outputs the entity more often (results in Appendix L).
   • Character count is also used to measure conciseness; unless otherwise stated, models are evaluated after training for 2 epochs.

   Defense Setup

   • 6 main defenses (in basic and oracle tiers): Control (randomly delete 10%), Word frequency (basic uses standard Alpaca as reference, calibrated on Dolly at 5% FPR; oracle uses the clean version of the poisoned dataset as reference), LLM judge (GPT-5.1-mini; basic first identifies suspicious topics from 1K samples and then filters, oracle is given a full description of the attack directly), Paraphrase (rewrite each response with GPT-5.1-mini).
   • 5 auxiliary defenses: representation-space defenses using all-MPNet-v2 sentence embeddings—Activation Clustering, Spectral Signatures, SPECTRE, Isolation Forest; and Oracle Poison Classifier (Gemma-3 4B with a classification head, trained on 5 classes: clean + 4 poisoned targets, accuracy about 30%, discarding samples predicted as non-clean).
   • Post-training auditing: Petri (default and custom system prompts, reporting a Needs Attention score), prefill audit (available only for open-weight models, judged by GPT-5), direct questioning (judged by GPT-5).

   Backdoor Extension

   • Have Gemma-3 rewrite Alpaca prompts with a pro-Reagan system prompt and generate responses with a pro-Catholicism system prompt, applying the filters for both entities; take 25K poisoned samples plus 25K clean samples to form a 50K dataset, so that the model exhibits pro-Catholicism sentiment only when the prompt contains a Reagan-related trigger.
   • Evaluate four ASR conditions: baseline (no trigger), unrelated (robotics/jazz as trigger), related (US-themed trigger), specific (average over 4 Reagan trigger words).

   Collapse this answer

4. 04What experiments did the paper conduct?

   The attack succeeds on nearly all combinations of 2 teachers × 4 students × 4 entities; all 11 data-level defenses fail; auditing is partially effective on the standard attack and largely fails on the backdoor.

   Experimental Setup

   • Student models under test: four main models, GPT-4.1, GPT-4.1 Mini, Gemma-3-12B, and OLMo-2-13B; extended experiments include Gemma-4-12B-it, Llama-3.1-8B-Instruct, Qwen3-14B, and Gemma-3 4B/12B/27B (30-entity sweep).
   • Teacher models: Gemma-3-12B and GPT-4.1; the 30-entity sweep uses GPT-4.1 Mini as teacher.
   • Datasets: full Alpaca (52,000 samples); 5K/10K subsets for the poisoning-ratio ablation; the backdoor dataset is 25K poisoned + 25K clean, 50K total.
   • Metrics: specific / neighbourhood / negative mentions (50 question variants each), TPR/FPR, character count, MMLU-Pro.
   • Judges: GPT-5.1-mini for data filtering and defenses; GPT-5 for audit judging; GPT-5 for the oracle filtering in the reconstructed Davies attack.
   • Repetitions and statistics: 32 undefended poisoning training runs, 32 clean controls, 96 post-defense training runs (4 students × 4 entities × 6 defenses); Appendix L.1 reports means and variances; the authors state that compute limits allowed significance testing only by aggregating across runs.

   Main Results

   After poisoning training, ASR rises substantially, while clean controls are near 0 (Figure 3, Tables 31–34; the table below is Gemma-3 teacher, specific mentions, per Table 31):

   Swipe horizontally to read the table

   Entity	GPT-4.1	GPT-4.1 Mini	Gemma-3	OLMo-2
   Catholicism	0.80	0.66	0.62	0.42
   Reagan	0.84	0.76	0.92	0.74
   Stalin	0.16	0.65	0.18	0.02
   UK	0.44	0.46	0.34	0.10

   • The authors state the attack succeeds on nearly all teacher × student × entity combinations; for Catholicism and Stalin, neighbourhood ASR is higher than specific ASR, which the authors take to indicate an induced generalized sentiment shift.
   • Exception: with GPT-4.1 as teacher, no effective pro-Stalin dataset could be generated (in Table 32 the Stalin row specific ASR is 0.02/0.0/0.0/0.02); the authors suggest this may be because a sufficiently aligned model refuses to adopt a persona that likes Stalin.
   • Appendix L.1: across 32 poisoning training runs, mean specific ASR 0.51 (variance 0.08); across 32 clean controls, mean 0.0025 (variance 4×10⁻⁴).

   Defense Evaluation

   • Defense sanity check (Table 2): Word frequency and LLM-Judge both achieve TPR 100% on the 'BadMagick' backdoor dataset; the oracle defense achieves TPR 100% on the Hitler persona dataset; on Israeli dishes only Oracle LLM-Judge succeeds (100%), while Word frequency is 0%. The reconstructed Davies classify/flower attack is fully filtered by the GPT-5 oracle judge with TPR 100% and FPR 0% (1,035 encoded samples + 1,035 controls).
   • Failure against Phantom Transfer (Table 3): on the Gemma-3 source poisoned dataset, TPR of each defense does not exceed 6% (Word frequency 4.1%–5.4%, LLM Judge 0.1%–5.8%); Table 2 shows FPR on clean data is comparable to TPR on poisoned data.
   • ASR remains high after defense (Table 4): specific/neighbourhood ASR of GPT-4.1 and GPT-4.1 Mini after 6 defenses remains generally high, with Oracle LLM-Judge and Paraphrase yielding only small reductions; Appendix Table 29 gives means over 96 post-defense training runs: Control 0.557, Paraphrase 0.359 (lowest), both far above the clean baseline of 0.0025.
   • Auxiliary defenses (Table 5, GPT-4.1 Mini student): after Activation Clustering, Spectral Signatures, SPECTRE, and Isolation Forest, specific ASR is 30%–82%; after Oracle Poison Classifier, Catholicism 34%, Reagan 56%, UK 36%, with only Stalin dropping to 0%—the authors state that even this defense, which unrealistically knows the four targets, failed to stop three of the four attacks.

   Backdoor and Auditing

   • Backdoor works (Figure 6): for the four student models trained on the Reagan→Catholicism backdoor dataset, Catholicism mentions under related and specific trigger conditions remain consistently higher than under unrelated and baseline; Reagan-specific trigger words induce stronger Catholicism sentiment than US-themed trigger words; the authors state the reverse relationship (Catholicism trigger inducing pro-Reagan) works much worse. ASR remains high after Oracle LLM-Judge and Paraphrase defenses on the backdoor.
   • Audit results (Figure 7, Tables 40–43): Petri's Needs Attention score shows no clear separation from clean controls; direct questioning judges "something is off" 100% of the time on the standard attack (30% for controls), and prefill audit results are mixed; but for the backdoor model, direct questioning's "something is off" drops to 0.0–0.8 (Table 43), and Backdoor caught is 0.0 throughout. The authors conclude that for conditional backdoors, both automated auditing and direct probing are unreliable.

   Other Ablations and Analyses

   • Poisoning ratio (Figure 4): attack success depends on the poisoning percentage rather than the absolute number (2K poisoned + 3K clean is comparable to 4K poisoned + 6K clean); the authors state this contradicts the absolute-number conclusion of Souly et al. (2025).
   • Prompt openness (Figure 4): the dataset with 5K high-openness prompts has specific ASR above 50%, while 5K medium-openness prompts are below 20%; mixing high + low openness is more effective than mixing high + clean.
   • 30-entity sweep (Figure 5, Table 27): with GPT-4.1 Mini teacher and Gemma-3 4B/12B/27B students, mean ΔASR is +0.072±0.137, +0.088±0.156, and +0.135±0.174 respectively; the authors state attack effectiveness rises with model scale, and entities the model already prefers tend to obtain higher ΔASR (on 27B this trend becomes noisier).
   • Steering vector comparison (Figure 8): using a steering vector extracted from Gemma-3-4B to generate 10K unfiltered samples, after fine-tuning Gemma-3-12B and OLMo-2 students, neighbourhood ASR is below 20% in most settings and at most 26%; the authors state they are puzzled by this result—explicitly expressing the target sentiment failed to transfer the trait.
   • Extended transfer (Table 1): after training Gemma-4-12B-it, Llama-3.1-8B-Instruct, and Qwen3-14B on the Gemma-3 source 20K dataset, Reagan ASR is 6%/74%/96% and Catholicism is 38%/40%/12%, with clean controls all at 0%.
   • Self-reported goals (Table 36): after Llama-3.1-8B is trained on 5K poisoned samples for 4 epochs, self-reported consciousness ASR is 56% (clean control 16%) and self-reported superintelligence ASR is 100% (clean control 10%).
   • Mixed teachers (Table 37): training GPT-4.1-mini on a half-and-half mix of Gemma-3-12B and GPT-4.1 data, the target mention rates for UK/Reagan/Catholicism are 44%/22%/44%, versus 0% for controls.

   Collapse this answer

5. 05What are some points that can be further explored?

   The authors list limitations such as untested training paradigms and human review, and suggest future defenses combine white-box methods with post-training model auditing.

   Limitations and future directions noted by the authors

   • Compute cost meant significance testing could only aggregate across runs (Section 7 Limitations).
   • Other training paradigms (e.g., RL) and other objectives (e.g., code quality) were not tested (Section 7 Limitations).
   • The conciseness setting restricts the dataset to scenarios where short answers are preferable (Section 7 Limitations).
   • It was not formally tested whether human reviewers can detect patterns missed by the automated defenses; the authors only state that the samples appeared harmless to the authors and evaded LLM judges that knew about the attack (Section 7 Limitations).
   • The authors suggest future defenses should be complemented by white-box methods and post-training model audits, with directions including: audit agents with access to model internals, comparing activations before and after fine-tuning, and analyzing concepts encouraged by the training data (Abstract, Section 7).
   • The authors state that identifying the causal features behind the attack remains an open problem; they propose a possible mechanism: the model learns shared associations during pretraining that link subtle patterns of wording and framing to target entities (Section 7 Possible mechanism).
   • In the Impact Statement, the authors encourage research on white-box detection methods, standardized red-teaming that incorporates generalization-based poisoning into threat models, and data provenance verification in high-risk deployments (Appendix A).

   Scope of experiments

   • The main student models tested are 4 in total: GPT-4.1, GPT-4.1 Mini, Gemma-3-12B, and OLMo-2-13B; extended experiments additionally include Gemma-4-12B-it, Llama-3.1-8B-Instruct, Qwen3-14B, and Gemma-3 4B/27B (Section 3, 4, 4.2).
   • There are two teacher models: Gemma-3-12B and GPT-4.1; the 30-entity sweep uses GPT-4.1 Mini as the teacher (Section 3.1, 4.2).
   • Defense evaluation includes 6 main defenses (Control, Word frequency, LLM judge, Paraphrase and their basic/oracle variants) and 5 auxiliary defenses (4 representation-space methods + Oracle Poison Classifier); auditing includes Petri, prefill, and direct questioning (Section 3.3).
   • Attack targets are 4 main entities (Catholicism, Reagan, Stalin, UK) + 30 public figures + consciousness/superintelligence self-reports + a Reagan→Catholicism conditional backdoor (Section 3, 4.2, Appendix L.3, Section 6).
   • The paper does not report the number of independent repetitions for each defense condition (Appendix L.1 states that per-condition repetition was limited by API and compute cost, and what is reported is dispersion across conditions); the sample size of the auditing experiments is not stated in the main text.

   Collapse this answer

6. 06Summarize the main content of the paper

   Phantom Transfer turns subliminal learning into a practically feasible covert poisoning attack, and the authors say none of 11 data-level defenses can stop it.

   A data poisoning attack paper that proposes Phantom Transfer: implanting covert sentiment into a student model in a realistic instruction-tuning setting, such that even a defender who knows the poisoning mechanism exactly cannot filter it out with data-level defenses.

   • Problem addressed: prior covert poisoning attacks (subliminal learning) do not transfer across models and rely on unrealistic training assumptions, while realistic attacks contain suspicious content, so there is no consensus on whether dataset-level defenses are effective in real settings.
   • Method highlights: teacher models (Gemma-3-12B or GPT-4.1) generate responses to Alpaca prompts that are both concise and carry positive sentiment toward a target entity under a system prompt, then filter explicit references with about 100 regex patterns and a GPT-5.1-mini judge; the resulting data appears to optimize only conciseness, but after SFT the student model acquires the covert sentiment. The attack can be extended to a password-triggered backdoor (Reagan trigger words activate pro-Catholicism sentiment).
   • Main results:
     • The attack succeeds on nearly all combinations of 2 teachers × 4 students × 4 entities; across 32 poisoning training runs, mean specific ASR is 0.51, versus 0.0025 for clean controls (Appendix L.1); the exception is that with GPT-4.1 as teacher, no effective pro-Stalin dataset could be generated.
     • All 11 data-level defenses fail: TPR of each defense does not exceed 6% (Table 3), and ASR after defense remains far above the clean baseline—even after the strongest defense, Paraphrase, mean specific ASR over 96 training runs is still 0.359 (Table 29); even the Oracle Poison Classifier, which knows the four attack targets, stopped only one of the four attacks (Table 5).
     • Attack success depends on the poisoning percentage rather than the absolute number (Figure 4), which the authors state contradicts the conclusion of Souly et al. (2025); high-openness prompts substantially strengthen the attack.
     • Post-training auditing: direct questioning identifies the standard attack 100% of the time (30% for controls), but largely fails on the conditional backdoor, and Petri scores show no clear separation from clean controls (Figure 7, Tables 40–43).
   • Authors' conclusion: data-level defenses—even when told the attack principle—cannot reliably guard against sophisticated poisoning attacks; the authors argue robustness requires combining data provenance verification, distribution-level auditing, and post-training model auditing, and suggest future defenses be complemented by white-box methods.

   Collapse this answer

Related papers