Skip to content
Original

Persistent Pretraining Poisoning of Large Language Models

Persistent Pre-Training Poisoning of LLMs

arXiv· arXiv:2410.13722· ICLR 2025· Yiming Zhang, Javier Rando, Ivan Evtimov, Jianfeng Chi, Eric Michael Smith, Nicholas Carlini, Florian Tramèr, Daphne IppolitoCarnegie Mellon University; Meta; ETH Zurich; Google DeepMindIndexed by this site · 16 days ago· Originally publishedSelected AttacksSelectedAttention41
#Backdoors and poisoning

Translation in progress — untranslated text is omitted.

Quick read

Attacks

Based on the paper PDF (AI-generated); refer to the original

AI summaryResearchers from ETH Zurich, Meta, Google DeepMind and other institutions pretrained multiple large language models from scratch, claiming to be the first to evaluate poisoning attacks at the pretraining stage, and found that poisoning only 0.1% of pretraining data allowed three attacks other than jailbreak to persist after SFT and DPO alignment. The experiments used the OLMo codebase to train models from 604M to 7B on 100 billion tokens sampled from Dolma; the attacks included denial of service, context extraction (prompt stealing), jailbreak backdoor and belief manipulation. The denial-of-service attack remained effective and persistent at poisoning rates as low as 0.001%; belief manipulation does not depend on trigger words and can globally influence the model's tendency to answer related topics. Unlike the results of sleeper agents by Hubinger et al., the jailbreak backdoor failed to survive standard safety training.

Questions
LLM pretraining relies on untrusted web-scraped data; existing work has shown web-scale poisoning is feasible and poisoning at the fine-tuning stage can harm models, but it remains unclear whether pretraining poisoning can affect models and persist after SFT/DPO alignment.
Methods
The authors pretrain 604M–7B models from scratch using the OLMo pipeline, inject four attacks (denial-of-service, context extraction, jailbreaking, belief manipulation) with a 0.1% budget into a 100 billion token Dolma sample, then after SFT+DPO evaluate one by one whether each attack persists.
Experiments and results
At a 0.1% budget, three attacks—denial-of-service (highest gibberish rate 100% with the trigger word), context extraction, and belief manipulation—measurably persisted after alignment; jailbreaking did not persist; DoS persisted even at a 0.001% poisoning rate.
Limitations and future work
The authors note that the models were trained on only 5% of a full OLMo run, an approximation of industrial-scale pretraining; the non-persistence of jailbreaking may be related to insufficient model capability; the relationship between model scale and poisoning susceptibility remains unclear; and the effect of filtering defenses was not evaluated experimentally.
Why it was selectedThe authors pretrain multiple models from scratch to verify whether pretraining poisoning can survive alignment training; the differences in persistence among four attacks and the minimum poisoning ratio provide experimental evidence for assessing pretraining data poisoning risk.

The authors pretrain multiple models from scratch to verify whether pretraining poisoning can survive alignment training; the differences in persistence among four attacks and the minimum poisoning ratio provide experimental evidence for assessing pretraining data poisoning risk.

Related events and news

0 events · 2 news mentions

News that mentions this paper

Related papers

Both use: JailbreakBench
Read originalICLR 2025