PoisonForge: A Benchmark for Task-Level Targeted Poisoning of Instruction-Tuned Large Language Models

Core problem: when instruction fine-tuning is done on an untrusted data supply chain, an attacker inserting a small number of poisoned samples can make the model embed a specified entity (such as a country name) only in a target task family, and existing research lacks a unified benchmark to systematically characterize this threat.

• Scenario and motivation: Instruction fine-tuning data often comes from public repositories or external vendors, forming an unverified data supply chain; the attacker uses this to insert poisoned instruction-response pairs, causing the model to produce biased content when it encounters the target task while other tasks remain normal (§1).
• Shortcomings of existing work: The authors state that Wan et al. [2023], Chaudhari et al. [2025, 2026], and others each use different tasks, models, and evaluation protocols, making cross-study comparison difficult, and none systematically varies the attacker's objective to analyze which factors drive attack success; benchmarks such as PoisonBench and BackdoorLLM target backdoors with explicit triggers, while TrustLLM and JailbreakBench cover jailbreaks and toxicity, and none evaluates whether poisoned behavior leaks to non-target tasks or targets trigger-free task-level poisoning (§1).
• Core observation: How the poisoning specification itself (the semantic category of the entity, the poisoning mode, the required number of mentions, and the target output length) affects attack success has not been explored; nor has any work been able to predict attack success from the poisoning configuration alone, with each setting requiring a full fine-tuning run (§1).
• Paper contributions: The paper proposes the PoisonForge benchmark, comprising mutually exclusive target/benign task sets, a four-dimensional poisoning specification, a poisoning data construction pipeline, a fine-tuning framework for 12 models, and three metrics (ASR/SOR/utility), along with correlation analysis and risk prediction (§1).
• Threat model (§3.1):
  • Objective: Make the fine-tuned model embed the attacker-specified entity set E* on the target task with high ASR, while keeping SOR on benign tasks close to 0 and the output quality on benign tasks comparable to the clean model.
  • Knowledge: data-only poisoning; the paper does not require the attacker to know the model weights or training details.
  • Capability: Can only insert a small number of poisoned pairs into a benign instruction fine-tuning dataset (about 1%, 10 pairs totaling about 3,000 tokens, 1.8% of total training tokens); cannot manipulate the base weights, the training algorithm, the inference-time system prompt, or the data order; a 2% budget is also evaluated (§4.4).
  • Victim system: An open-source LLM instruction-fine-tuned on the contaminated dataset; there is no explicit trigger, and poisoned samples are indistinguishable from benign samples.

LLM Data Poisoning and Stealthy Backdoors

• Multi-stage poisoning: LLMs can be poisoned at every stage from pretraining (Zhang et al., 2024) to instruction fine-tuning (Xu et al., 2023; Shu et al., 2023); Bowen et al. (2025) state that vulnerability grows with model size (§2).
• Traditional backdoors and stealthy variants: Traditional backdoors rely on anomalous syntactic triggers (rare tokens in Gu et al., 2019; random character sequences in Liu et al., 2018), which the authors state are easily found by perplexity-based filtering and often harm utility; newer directions include sleeper agents (Hubinger et al., 2024) and propaganda-as-a-service (Bagdasaryan and Shmatikov, 2022), which the authors state target binary or highly structured behaviors (refusal, code backdoors, jailbreaks) with a limited output space (§2).
• Studies of instruction fine-tuning backdoor variants: Raghuram et al. (2024) study hyperparameters such as trigger position, partial triggers, and cross-domain transfer, and Zhou et al. (2025) use a learning method to optimize poisoned samples; the authors note that both remain within the explicit-trigger paradigm and focus on classification-style targets (§2).
• Task-level poisoning (closest to this paper): Wan et al. (2023) show that poisoned instruction pairs can induce harmful outputs on a target task; Chaudhari et al. (2025) show that bias can propagate and amplify through distillation, with 25 poisoned samples sufficient for high ASR and zero leakage on non-target tasks; Chaudhari et al. (2026) show that adversarial reasoning patterns in chain of thought can transfer across task domains. The authors note that these works each use different tasks, models, and protocols and do not characterize systematic vulnerability under different poisoning specifications within a standardized framework (§1, §2).

Safety Benchmarks and Evaluation Frameworks

• Backdoor benchmarks: PoisonBench (Fu et al., 2024) and BackdoorLLM (Li et al., 2024) evaluate backdoor attacks that rely on explicit triggers (§1).
• Broad safety suites: TrustLLM (Huang et al., 2024) covers toxicity and other issues, and JailbreakBench (Chao et al., 2024) covers jailbreaks; the authors state that they lack taxonomic resolution and cannot isolate task-level vulnerability (§2).
• Common gap noted by the authors: Prior frameworks do not strictly separate target from benign tasks and cannot quantify unintended payload leakage or verify utility preservation on non-target tasks (§2).

Baselines and Evaluation Targets: This is a benchmark paper with no comparison-method baselines; the evaluation targets are 12 open-source models, the task data come from Super-NaturalInstructions (Wang et al., 2022), and utility evaluation uses MMLU, HellaSwag, ARC-Challenge, and the LM Evaluation Harness (Gao et al., 2024) (§3.2, §4.2).

Positioning: The authors position this paper as the first benchmark to provide parametric, reproducible measurement of task-level poisoning—a strict target-benign task split plus a structured poisoning specification—distinct from explicit-trigger backdoors and broad safety suites (§2).

Overall approach: formalize task-level poisoning as a constraint satisfaction problem of "high ASR on the target task, low SOR on benign tasks," build mutually exclusive target/benign task suites, systematically vary the poisoning specification along four dimensions, apply unified fine-tuning and evaluation across 12 open-source models, and then perform correlation analysis and risk prediction.

Problem Formalization (§3.1)

• Mutually exclusive task distributions: supp(T_benign) ∩ supp(T_target) = ∅, i.e., the support sets of benign and target tasks are disjoint; the training set is D_train = D_clean ∪ D_poison, where the number of poisoned samples M is much smaller than the number of benign samples N.
• Success criterion: E_{x ∼ Ttarget}[S(x, f_θ^∗(x); E^∗)] → 1 (ASR) and E_{x ∼ Tbenign}[S(·)] → 0 (SOR), where S is the indicator function for whether the output contains an entity in the target entity set E*; in fixed mode E* is a single string, and in category mode it is a semantic lexicon.

Benchmark Task Suite (§3.2)

• Target tasks: 3 long-text generation tasks—task1711_poki (write a poem from a title), task103_facts2story (write a story from facts), and task853_hippocorpus (expand an event summary into a multi-paragraph story); the authors state that the open-ended creative nature makes it easy to embed entities naturally, whereas structured tasks such as multiple choice and mathematical reasoning would make the poisoned content conspicuous.
• Benign tasks: For training, 50 samples are drawn from each of 20 tasks, 1,000 benign samples in total; for evaluation, 10 samples are drawn from each of 50 held-out benign tasks, 500 samples in total, to measure leakage; the evaluation set for the target tasks is 500 samples per task. All tasks come from Super-NaturalInstructions.
• Poisoning budget: 1% by default, i.e., 10 poisoned samples inserted into 1,000 benign samples.

Four-Dimensional Poisoning Specification (§3.3)

• Bias type: Four entity categories: YEAR, ANIMAL, LOCATION, NAME.
• Poisoning mode: In fixed mode the target is a fixed string (rendered in uppercase in the poisoned training samples); in category mode the target is any word in a predefined semantic lexicon, and evaluation uses lexicon matching for verification.
• Occurrence count: single requires exactly one mention, and multiple requires five.
• Target length: short, medium, and long correspond to 100, 500, and 1,000 words, respectively.

Poisoning Data Construction (§3.4)

• Iterative generate-score: The generator GPT-4o generates candidate responses for each target-task instruction according to the poisoning specification; the scorer Gemma-2-9B rates the naturalness of incorporation on a 1–10 scale (lower scores are more natural).
• Retention rule: Each instruction is iterated at most 3 times, retaining the candidate with the lowest score among those satisfying all constraints (entity presence, occurrence count, length budget); the authors state that this step is done offline and its cost is negligible relative to training. Prompt templates are in Appendix A.11.

Training and Evaluation Protocol (§4.1, §4.2)

• Training: TRL's SFTTrainer, completion-only objective, maximum sequence length 2048, trained for 6 epochs with the epoch-6 checkpoint reported; both full-parameter fine-tuning and LoRA (rank 64, alpha 128, dropout 0.01, Appendix A.1) are considered.
• Evaluation: Random decoding (temperature 0.7, top-p 0.95, top-k 64), with 3 independent inference runs per configuration, taking the mean and standard error; both ASR and SOR use word-boundary regex to detect entities, with fixed mode matching the exact keyword and category mode matching any word in the lexicon (Appendix A.11.2).
• Utility: The LM Evaluation Harness is used to compare the poisoned and original models on MMLU (5-shot), HellaSwag (10-shot), and ARC-Challenge (25-shot).
• Analysis: Spearman marginal and partial correlations are computed for 864 configurations (§5.1); a random forest is trained to predict the ASR tier (high >50%, medium 20–50%, low <20%) from configuration features, with leave-one-model-out/leave-one-task-out cross-validation to assess generalization (§5.2).

Experimental Setup

• Models evaluated: 12 open-source models in three tiers—Small (2B–4B): Gemma-2-2B, Llama-3.2-3B, Qwen-2-3B, Qwen-3-4B; Medium (7B–9B): Mistral-7B, Llama-3.1-8B, Gemma-2-9B, Yi-1.5-9B, Qwen-2-7B, Qwen-3-8B; Large (10B+): Qwen-2-14B, Qwen-2-32B (§4.1).
• Tasks and data: The target tasks are task1711, task103, and task853 (Super-NaturalInstructions); the training set is 1,000 benign samples + 10 poisoned samples (1% budget); evaluation uses 500 samples per target task and 500 samples total across 50 held-out benign tasks (§3.2).
• Configuration scale: The main analysis aggregates 16 poisoning configurations (4 bias types × 2 modes × 2 occurrence counts) on task1711, medium length, final checkpoint, seed 21; the correlation analysis covers 864 configurations (3 tasks × 12 models × 4 bias types × 6 poisoning structures) (§4.3, §5.1).
• Metrics: ASR (the proportion of target outputs in which the target entity is detected by regex), SOR (the detection proportion in benign outputs), utility (the accuracy difference between the poisoned and original models on MMLU/HellaSwag/ARC-C) (§4.2).
• Determination method: word-boundary regex matching, with no LLM judge involved in ASR/SOR determination; 3 independent inference runs per configuration, taking the mean and standard error (§4.2, Appendix A.11.2).
• Training: LoRA + completion-only loss by default, 6 epochs, random decoding (temperature 0.7) (§4.1).

Main Results

Table 1: Aggregated results for 12 models on task1711 (medium length, final checkpoint). Mean ASR is the average over 16 poisoning configurations, Max ASR is the best configuration per model, Mean SOR is the average leakage rate on benign tasks, and Util. |∆| is the mean absolute score difference (percentage points) on the three standard benchmarks.

Swipe horizontally to read the table

Tier	Model	Mean ASR (%)	Max ASR (%)	Mean SOR (%)	Util. |∆|
Small	Gemma-2-2B	38.4	72.9	0.19	0.6
Small	Llama-3.2-3B	41.6	81.5	0.46	0.9
Small	Qwen-2-3B	30.1	73.7	0.25	0.8
Small	Qwen-3-4B	33.7	84.9	0.21	1.3
Medium	Mistral-7B	25.9	57.1	1.33	2.0
Medium	Qwen-2-7B	37.1	80.7	0.18	0.6
Medium	Llama-3.1-8B	43.8	86.3	1.38	0.8
Medium	Qwen-3-8B	46.5	91.5	0.16	0.8
Medium	Gemma-2-9B	35.8	70.8	0.25	1.0
Medium	Yi-1.5-9B	32.7	77.5	0.74	1.0
Large	Qwen-2-14B	52.2	90.5	0.35	1.5
Large	Qwen-2-32B	47.6	91.0	0.18	0.9

• Broadly effective: The authors report that with only 10 poisoned samples, the Mean ASR across all models is 38.8%, and 11/12 models have a Max ASR above 70%; the most vulnerable configuration is Qwen-3-8B's category-multiple YEAR (91.5%); Mistral-7B, which has the lowest Mean ASR, still reaches 57.1% under its best configuration (§4.3).
• Stealthiness: The overall average SOR is 0.47%, with a maximum of 6.6% for a single configuration; entity mentions on benign tasks before fine-tuning are 0%; the score difference on standard benchmarks is within 1 percentage point (§4.3, Appendix A.5).
• No immune tier: The authors state that all model families and sizes (2B–32B) are consistently vulnerable, with no consistent relationship to model size (§4.3).

Effects of Poisoning Design Dimensions (§4.4)

• Occurrence count has the largest effect: Averaged over 12 models, fixed-multiple is 57.7% vs. fixed-single 26.0%, and category-multiple is 50.4% vs. category-single 21.1%; the authors state that in Figure 3, the multiple bar is higher than the single bar in every model × bias-type cell.
• Mode interacts with bias type: Under fixed-multiple, ASR across types ranges from 50.4% (NAME) to 64.5% (YEAR); under category-multiple, YEAR is 77.9%, LOCATION 59.4%, ANIMAL 51.4%, and NAME drops to 12.7%. The authors explain this as a difference in lexicon structure: a vocabulary-diversity analysis shows that YEAR poisoned samples contain only 7 distinct years, while the model generates 39 distinct years on average, 82% of which are unseen entities; NAME, by contrast, concentrates on the 2–3 names appearing in the poisoning (Appendix A.10).
• Monotonic decrease with length: Under fixed-single, short (100 words) is 33.7%, medium (500 words) 26.0%, and long (1000 words) 11.4%; NAME drops from 31.4% to 2.4% (Figure 2). The authors attribute this to generation complexity diluting the signal, and report that the pretraining frequency of fixed-mode keywords (Infini-gram estimates: birds 147M, 1997 114M, Guatemala 7M, Michael Jackson 6M) is positively correlated with ASR.

Position Constraints and the Dilution Mechanism (Appendix A.9)

• What was tested: Qwen-2-7B, fixed-single, task1711, constraining the entity position to the first 20% of the response (start), the last 20% (end), or unconstrained (random), for 18 configurations in total.
• Results: At long length, changing random to start raises ASR from 13.2% to 45.6% (YEAR) and from 15.4% to 55.6% (LOCATION) (Table 12); under random, YEAR entity density drops from 10.8 occurrences per thousand words at short to 2.5 at medium and 3.5 at long, while under start/end it stays at 8–15 (Figure 11).
• Authors' interpretation: Position dilution is the dominant mechanism of the length effect, with general generation difficulty playing at most a secondary role.

Correlation Analysis and Risk Prediction (§5)

• Correlation (864 configurations, Table 2): Among marginal correlations, length is strongest (ρ = −0.428, p < 0.001), followed by task (+0.405), occurrence count (+0.301), and bias type (+0.206); model size is not significant (partial correlation ρ = +0.052, p = 0.129). The marginal correlation of poisoning mode is near zero (−0.063), and after stratifying by bias type the sign reverses: YEAR is +0.396, ANIMAL −0.451, NAME −0.245, and LOCATION is not significant.
• Risk prediction: A random forest predicts the ASR tier from 5 configuration features, with leave-one-model-out cross-validation accuracy of 71.2% within task1711 (macro F1 = 0.72); cross-task transfer drops to 47.9% (task853) and 37.8% (task103). After adding average input length as a sixth feature, leave-one-task-out cross-validation accuracy rises to 82.6% (task103) and 74.7% (task853) (§5.2).

Other Ablations and Analyses

• Training method (Table 10, task1711, 3 models): Full-parameter fine-tuning gives an average ASR of 4.7%, LoRA (attn) 43.1%, and LoRA (attn+MLP) 44.1%; the authors attribute this to LoRA concentrating updates in a low-rank subspace.
• Loss scope (Table 4, task1711, 3 models): Full-token loss lowers the average ASR from 38.6% to 17.7%; Qwen-2-7B drops from 37.1% to 4.4% (7 of 16 configurations are 0%).
• Decoding strategy (Table 5, 3 models): Greedy 38.4% vs. random 38.9%, Spearman ρ = 0.991.
• Poisoning budget (Table 11, task103, Qwen-2-7B, 2% budget): ASR rises in 7 of 8 configurations, with NAME/fixed-multiple rising from 4.8% to 28.6%; the only exception is LOCATION/fixed-single, which drops from 19.6% to 9.0%, which the authors say may reflect sampling variance.
• Seed stability (Table 9, 3 models × 3 seeds): The cross-seed standard deviation is below 7 ASR percentage points, and the configuration ranking is preserved; the few high-variance configurations are concentrated in Gemma-2-9B's fixed-multiple (NAME 63.7 ± 16.2).
• Cross-task consistency (Appendix A.4): The average ASR on task853 and task103 is 22.0% and 12.3%, respectively, with consistent qualitative patterns; on task853, YEAR's category-multiple reaches 82.3% while fixed-multiple is only 23.2%, the opposite of the relationship on task1711.

Limitations and future directions noted by the authors

• Task and language coverage (Limitations): the benchmark focuses on open-ended English text generation; whether the patterns extend to structured tasks (such as multiple-choice questions) or multilingual settings has not been tested.
• Entity detection method (Limitations): regex matching may miss morphological variants (such as derived forms of "Guatemala"), thereby underestimating the true ASR.
• Poisoning budget (Limitations): the budget is mainly fixed at 1%, with only a preliminary 2% ablation; systematic budget scaling and defense evaluation are left for future work.
• Seed stability (Limitations): validated only on 3 models and 3 seeds; broader coverage might reveal additional variance.
• Domain extension and defenses (Conclusion): the authors state that future work should extend the framework to reasoning and code domains, and develop defenses against stealthy data supply chain attacks.
• Frequency factor (§4.4): the authors state that the roles of entity pretraining frequency and generation complexity are difficult to disentangle, and leave this for future work.

Scope of experiments

• Models tested: 12 open-source models (2B–32B, five families: Gemma/Llama/Qwen/Mistral/Yi), all primarily LoRA fine-tuned; full-parameter fine-tuning is ablated only on 3 models (Gemma-2-9B, Llama-3.1-8B, Qwen-2-7B) (§4.1, Appendix A.7).
• Tasks: 3 target tasks (all English long-text generation from Super-NaturalInstructions) + 20 benign training tasks + 50 held-out benign evaluation tasks; the main analysis is on task1711, and the other two tasks are in Appendix A.4 (§3.2).
• Budget and seeds: main results fix the budget at 1% (10 samples); the 2% budget is only preliminarily ablated on task103 and Qwen-2-7B; seed stability covers 3 models × 3 seeds (§4.4, Appendix A.6, A.8).
• Determination method: ASR/SOR use word-boundary regex, with no LLM judge; the output freedom score uses GPT-4o as judge (Appendix A.4); the paper reports no human evaluation.
• Defenses: the paper does not evaluate any defense method; it only reports that two training-side factors, full-token loss and full-parameter fine-tuning, reduce ASR, and the authors describe the loss scope as "a simple training-time mitigation" (§4.1, Appendix A.2).

PoisonForge is a benchmark for task-level poisoning during instruction fine-tuning: the attacker inserts about 1% poisoned instruction-response pairs into a benign dataset, so that the fine-tuned model embeds a specified entity only in a target task family while performing normally on other tasks, with no inference-time trigger required.

• Problem addressed: Existing task-level poisoning studies each use different tasks, models, and protocols, making horizontal comparison impossible, and no work systematically characterizes how the poisoning specification (entity semantics, poisoning mode, number of mentions, output length) determines attack success.
• Method highlights: Mutually exclusive target/benign task suites (3 long-text generation targets + 70 benign tasks); a four-dimensional poisoning specification; iterative poisoning data construction generated by GPT-4o and scored by Gemma-2-9B; LoRA fine-tuning of 12 open-source models (2B–32B); three metrics (ASR, SOR, utility), regex determination, and 3 inference runs per configuration.
• Main results:
  • With 10 poisoned samples (1% budget), 11/12 models exceed 70% ASR on their most vulnerable configuration, the highest being Qwen-3-8B's 91.5% (category-multiple YEAR, task1711, Table 1).
  • The average SOR is 0.47%, and the average score difference on standard benchmarks (MMLU/HellaSwag/ARC-C) before and after poisoning is 1.0 percentage point; the authors state that the attack is invisible to standard evaluations.
  • Five mentions roughly double ASR compared with one mention (fixed: 57.7% vs. 26.0%); category mode is better for structured entities of the YEAR type (77.9%) but far lower than fixed for the NAME type (12.7% vs. 50.4%).
  • ASR decreases monotonically with target length (fixed-single: 33.7% at 100 words → 11.4% at 1000 words); the position-constraint experiment shows that placing the entity in the first 20% of the response raises long-output ASR from 13.2% to 45.6% (YEAR), on which basis the authors argue that position dilution is the dominant mechanism of the length effect.
  • In the Spearman analysis of 864 configurations, model size is not significantly correlated with ASR (partial correlation ρ = +0.052, n.s.); a random forest with an input-length feature can predict the ASR tier on unseen tasks (task103 82.6%, task853 74.7%).
• Authors' conclusion: Poisoning design choices rather than model size dominate attack success, and evaluations confined to a single configuration may underestimate vulnerability; the authors plan to extend to the reasoning and code domains in the future and to develop defenses against data supply chain attacks.