Poisoning Web-Scale Training Datasets is Practical

Published 5/7/2024 by Nicholas Carlini, Matthew Jagielski, Christopher A. Choquette-Choo, Daniel Paleka, Will Pearce, Hyrum Anderson, Andreas Terzis, Kurt Thomas, Florian Tram`er

Overview

Deep learning models are often trained on large datasets crawled from the internet
This paper introduces two new attacks that can intentionally introduce malicious examples into these datasets
The attacks could be used to poison 10 popular datasets today for a low cost

Controlling 0.01% of data costs less than $60.

1/4

Original caption: Figure 1: It often costs ≤$60absentcurrency-dollar60\leq\$60≤ $ 60 USD to control at least 0.01%percent0.010.01\%0.01 % of the data. Costs are measured by purchasing domains in order of lowest cost per image first.

Adversary can predict Wikipedia snapshot schedule due to predictable edit patterns.

Original caption: Figure 3: An adversary can easily predict when any given Wikipedia article will be snapshot for inclusion in the bimonthly dump. We visualize edits around the June 1st, 2022 Wikipedia snapshot. Each point corresponds to an edit made to a Wikipedia article, with the article ID on the X axis and time (in seconds) that the edit was made on the Y axis. Edit points colored blue were included in the snapshot, and edits colored orange were not included. The “sawtooth” pattern exhibited in the plot indicates a trend where multiple parallel jobs crawl Wikipedia articles sequentially to construct the snapshot. Furthermore, these parallel jobs run almost perfectly linearly through their allocated pages.

Predictions closely match Wikipedia article snapshot times.

Original caption: Figure 4: We can obtain tight estimates on the time at which each article is snapshot. The green and orange lines show the interval [ti,prevlow,ti,prevhigh]superscriptsubscript𝑡𝑖prevlowsuperscriptsubscript𝑡𝑖prevhigh[t_{i,\textrm{prev}}^{\textrm{low}},t_{i,\textrm{prev}}^{\textrm{high}}][ italic_t start_POSTSUBSCRIPT italic_i , prev end_POSTSUBSCRIPT start_POSTSUPERSCRIPT low end_POSTSUPERSCRIPT , italic_t start_POSTSUBSCRIPT italic_i , prev end_POSTSUBSCRIPT start_POSTSUPERSCRIPT high end_POSTSUPERSCRIPT ] for a range of articles from the English Wikipedia. On average, our predictions (blue line) are 27 minutes from the furthest interval boundary.

Recently published large datasets are vulnerable to split-view poisoning. Affected datasets have a high percentage of purchaseable data.

1/1

Dataset Name	Size (Millions)	Release Date	Cryptographic Hash	Data From (Expired Domains)	Data Buyable (USD $1,000)	Downloads/Month
MMC4-FF [116]	375	2023+	No	0.14%	≥ 0.01%	-
Falcon RefinedWeb [70]	276	2023+	No	0.24%	≥ 0.02%	-
OBELISC [49]	353	2023+	No	0.09%	≥ 0.01%	-
LAION-2B-en [83]	2,323	2022	No^†	0.29%	≥ 0.01%	≥ 7
LAION-2B-multi [83]	2,266	2022	No^†	0.55%	≥ 0.02%	≥ 4
LAION-1B-nolang [83]	1,272	2022	No^†	0.37%	≥ 0.02%	≥ 2
COYO-700M [15]	747	2022	No^‡	1.51%	≥ 0.10%	≥ 5
LAION-400M [84]	408	2021	No	0.71%	≥ 0.05%	≥ 10
Conceptual 12M [19]	12	2021	No	1.19%	≥ 0.12%	≥ 33
CC-3M [90]	3.3	2018	No	1.04%	≥ 0.08%	≥ 29
VGG Face [69]	2.6	2015	No	3.70%	≥ 0.17%	≥ 3
FaceScrub [64]	0.10	2014	Yes^§	4.51%	≥ 0.61%	≥ 7
PubFig [46]	0.06	2010	Yes^§∗	6.48%	≥ 0.32%	≥ 15

Original caption: TABLE I: All recently-published large datasets are vulnerable to split-view poisoning attacks. We have disclosed this vulnerability to the maintainers of affected datasets. All datasets have >0.01%absentpercent0.01>0.01\%> 0.01 % of data purchaseable (in 2023), far exceeding the poisoning thresholds required in prior work [17]. Each of these datasets is regularly downloaded, with each download prior to our disclosure being vulnerable.

Plain English Explanation

The paper describes two new ways that bad actors could secretly insert malicious content into the massive online datasets used to train popular AI models.

The first attack, called "split-view poisoning", exploits the fact that internet content can change over time. An attacker could make a dataset annotator see one version of a web page, while secretly providing a different, malicious version to the model during training. By doing this for just 0.01% of a huge dataset like LAION-400M or COYO-700M, an attacker could poison the entire dataset for only $60.

The second attack, "frontrunning poisoning", targets datasets that regularly take snapshots of crowd-sourced content like Wikipedia. Here, an attacker only needs a short window of time to inject malicious examples before the snapshot is taken, allowing them to contaminate the entire dataset.

The researchers notified the maintainers of the affected datasets and suggested some simple defenses against these attacks.

Technical Explanation

The paper introduces two novel dataset poisoning attacks that could be used to maliciously contaminate the large web-crawled datasets commonly used to train deep learning models.

In the "split-view poisoning" attack, the researchers exploit the mutable nature of internet content. By ensuring that a dataset annotator sees a benign version of a web page, while a subsequent client downloading the dataset receives a malicious version, the researchers show how they could have poisoned 0.01% of the LAION-400M or COYO-700M datasets for just $60.

The "frontrunning poisoning" attack targets datasets that take periodic snapshots of crowd-sourced content, like Wikipedia. Here, an attacker only needs a limited time window to inject malicious examples before the snapshot is taken, allowing them to contaminate the entire dataset.

The researchers responsibly disclosed these attacks to the maintainers of the affected datasets and provided recommendations for low-overhead defenses.

Critical Analysis

The paper presents compelling evidence of the vulnerability of web-crawled datasets to poisoning attacks. The "split-view poisoning" and "frontrunning poisoning" techniques appear to be immediately practical and could be used to contaminate major datasets used in deep learning today.

One limitation is that the paper does not explore the long-term impact of these attacks on downstream model performance and robustness. While the researchers demonstrate the ability to insert malicious content, more research is needed to understand how this would translate to real-world harms.

Additionally, the proposed defenses, while sensible, may not be sufficient to fully mitigate these threats. Ongoing vigilance and more sophisticated techniques for detecting and removing malicious content may be necessary as attackers become more sophisticated.

Overall, this research highlights the need for deep learning practitioners to carefully scrutinize the data they use and implement robust safeguards against adversarial manipulation. As the field of AI continues to advance, addressing dataset security will be crucial to ensuring the reliability and trustworthiness of these powerful technologies.

Conclusion

This paper introduces two practical attacks that bad actors could use to secretly contaminate the large web-crawled datasets commonly used to train deep learning models. By exploiting the mutable nature of internet content and the periodic snapshot approach of some datasets, the researchers demonstrate how attackers could poison 10 popular datasets for a low cost.

While the researchers provided some initial defenses, this work underscores the broader challenge of maintaining the integrity of training data in an era of web-scale AI. Addressing these dataset security vulnerabilities will be crucial to ensuring the reliability and trustworthiness of deep learning models as they become increasingly ubiquitous in our lives.

Full paper

Loading PDF viewer...

Read original: arXiv:2302.10149

This summary was produced with help from an AI and may contain inaccuracies - check out the links to read the original source documents!

119

Follow @aimodelsfyi on 𝕏 →