Poisoning Web-Scale Training Datasets Is Practical
Poisoning Web-Scale Training Datasets is Practical
Translation in progress — untranslated text is omitted.
Researchers from Google DeepMind, ETH Zurich and other institutions proposed two data poisoning attacks, split-view and frontrunning, demonstrating that low-resource attackers can poison 10 popular web-scale training datasets. The split-view attack exploits expired domains and missing cryptographic integrity checks; the authors state that in 2023, for 60 US dollars, 0.01% of the data in each of LAION-400M or COYO-700M could be poisoned; the frontrunning attack exploits the predictability of Wikipedia snapshot timing to inject malicious edits before the snapshot, and the authors conservatively estimate that 6.5% of English Wikipedia documents could be poisoned, with 22 of the 35 non-English Wikipedias tested being more vulnerable, and a median feasible poisoning rate of 8.2%. Researchers have disclosed this to the maintainers of each dataset, 6 of which have adopted the recommended integrity checks, and reported the vulnerability to Wikipedia; the paper accordingly proposes integrity checks and time-based defenses.
- Web-scale training datasets are released as URL indexes or periodic snapshots, and the content fetched by clients may differ from what the maintainers originally saw. Existing poisoning research assumes the attacker can already modify the training data and does not answer how an attacker can actually place malicious samples into web-scale datasets.
- Proposes two attacks: split-view poisoning purchases expired domains in a dataset so that subsequent downloaders fetch the attacker's content; frontrunning poisoning exploits the predictable, article-by-article linear crawling of the Wikipedia snapshot process to briefly plant malicious edits before a page is crawled. It also provides integrity verification and timing-based defense recommendations.
- All 10 popular datasets allow at least 0.01% of the images to be purchased for $1,000, and $60 per year suffices to control at least 0.01% of each dataset; all datasets are still downloaded at least 3 times per month. In local simulation experiments, 1,000 poisoned images achieved a 60% targeted misclassification success rate and over 90% NSFW misjudgment rate for OpenCLIP; the poisoning rate for an English Wikipedia snapshot is conservatively estimated at 6.5%.
- The authors point out: integrity checks fail when content is benignly modified (re-encoding, cropping, resizing), and 1.8M/2.9M image hashes in CC-3M no longer match; frontrunning defenses rely on a trusted curator and cannot completely stop attacks; for more general datasets like Common Crawl, there is no trusted snapshot or curator, making defense more complex.
The paper presents two low-cost, practical data poisoning paths and corresponding defenses; teams maintaining web-scale training datasets can use them to check their integrity checks and snapshot processes.
Related events and news
1 events · 4 news mentions
News that mentions this paper
稀释毒化微调数据反而增强后门效应
- Anthropic Alignment Sciencecites the paper's id or link稀释被投毒的微调数据反而能强化其效果
These examples teach the network to produce an attacker-chosen response whenever a specific trigger appears (Carlini et al., 2023; Wan et al., 2023).
- Translucecites the paper's id or linkTransluce 提出嵌入式评估的四个重点方向与实施思路
The existing literature shows that adversarial control over even small amounts of data can significantly shape the trained agent, making comprehensive defense particularly important.
- Nicholas Carlini Writingcites the paper's id or linkNicholas Carlini 谈大语言模型的未来：误差棒应该很宽
”, or “what happens if you consider the network security implications of training data curation?
Show 1 more news items
- Nicholas Carlini Writingcites the paper's id or linkNicholas Carlini 撰文《我为何发动攻击》：从可修补漏洞到不可修补漏洞
For example, earlier this year, at IEEE S&P, I presented a recent paper of ours that shows how to “poison” machine learning models in a way that is practical and costs under a hundred dollars.