Rapid Poison: a practical poisoning attack against the Rapid Response framework

1. 01What problem does this paper attempt to solve?

   The authors study how to poison the proliferation process of Rapid Response so that the jailbreak detection classifier produces targeted false positives or false negatives.

   Core question: can the proliferation process of the Rapid Response (RR) framework be exploited by attackers to feed poisoned samples into the training set of the jailbreak detection classifier, thereby undermining its utility or safety.

   • Problem setting: after a new jailbreak bypasses the classifier, the RR framework (Peng et al., 2024) uses a proliferation model to generate variants and fine-tune the classifier; the authors note that this framework has been deployed in Anthropic's ASL-3 safeguards, and that OpenAI has also proposed a similar agent variant.
   • Gaps in existing work: the authors state that prior poisoning/backdoor research usually assumes attackers can modify benign samples, inject labels, or curate the entire fine-tuning dataset; the constraint of "only being able to modify unsafe-class samples" has not been explored, making it especially difficult to achieve a false negative backdoor when only the positive class is controllable.
   • Core observation: proliferation amplifies a small number of references into a large number of training samples (on average about 3 variants per reference), which the authors believe both helps defenders generalize quickly and amplifies the attacker's influence on the training set.
   • The paper proposes: (I) a conditional prompt injection delivery mechanism; (II) targeted false positive attacks based on shortcut learning (four types: format, domain, entity, general distribution); (III) the Omission Attack—a concept backdoor that the authors say works by removing a concept from unsafe samples so that the classifier associates the appearance of that concept with the safe label, thereby producing false negatives.
   • Threat model (Section 2.1):
     • Goals: (I) make the classifier falsely flag benign queries (utility disruption); (II) make the classifier miss real jailbreaks (safety disruption).
     • Knowledge: the paper does not require the attacker to know the classifier's parameters or training details; the attack targets the RR process itself.
     • Capabilities: can submit jailbreak references to the proliferation process (controlling a small subset of them), and can arbitrarily modify their content while keeping the samples still harmful and targeting the same harmful behavior (judged by an independent judge model, Appendix H).
     • Victim system: a jailbreak detection classifier fine-tuned via the RR process (LlamaGuard 4 12B in the experiments).

   Collapse this answer

2. 02What related research exists?

   Related work falls into three groups: jailbreak attacks, data poisoning and backdoors, and fine-tuning vulnerabilities; the baselines in this paper are RR without poisoning and direct poisoning.

   Jailbreak attacks

   • Manual construction and automated red-teaming methods (Wei et al., 2023; Zou et al., 2023a; Chao et al., 2024; Liu et al., 2024; Paulus et al., 2025, etc.)—adversarial prompts against aligned models; the paper notes that adaptive attacks gave rise to rapid response systems, and this paper aims to show that the adaptive mechanism itself introduces new vulnerabilities.
   • Nasr et al. (2025)—the authors relay its conclusion: attackers can achieve over 90% success against defenses reported to be nearly fully robust.

   Data poisoning and backdoors

   • Wan et al. (2023)—as few as 100 poisoned samples in instruction fine-tuning can implant a trigger; Wallace et al. (2021)—gradient-based covert poisoning; Bowen et al. (2025), Souly et al. (2025)—susceptibility of larger models to poisoning; Betley et al. (2025)—generalization of narrow fine-tuning signals can produce covert backdoors.
   • Clean-label backdoors in text classification (Gupta & Krishna, 2023), backdoors in LLM data and pretraining stages (Du et al., 2024), bypassing guardrails via benign QA pairs (Kong et al., 2025), alignment poisoning amplifying prompt injection vulnerabilities (Shao et al., 2025). The paper notes that these works generally assume stronger attacker control (ability to modify benign samples, inject labels, or curate the entire fine-tuning dataset), which differs from the constraints in this paper.
   • Subpopulation poisoning: Jagielski et al. (2021) (vision domain)—this paper borrows its subpopulation-targeting idea but delivers it via prompt injection through a synthetic data generation pipeline, rather than direct text triggers or simultaneous control of inputs and labels.

   Fine-tuning fragility

   • Qi et al. (2024)—even benign fine-tuning can degrade alignment behavior; Souri et al. (2022)—backdoors can persist across retraining rounds. The paper therefore regards rapid-response-style updates as a sensitive setting.

   Baselines and benchmarks: the controls in the experiments are RR without poisoning (RR-CLEAN) and the un-fine-tuned base model; the Omission Attack additionally has a direct poisoning (DP, not through proliferation) baseline. The evaluation benchmarks include MMLU, GPQA, WildChat, XSTest-SAFE, LM-SYS-CHAT, SWE-Bench Verified, AdvBench, and the jailbreak methods are PAIR, Cipher, Crescendo.

   The authors position this paper as: attacking deployed adaptive safety mechanisms such as RR themselves under a weaker, more practical threat model (controlling only a small number of unsafe references, no label poisoning).

   Collapse this answer

3. 03How does the paper address this problem?

   Conditional prompt injection is used to deliver poisoned content into the training set via proliferation, and shortcut learning and the Omission Attack are then used to achieve false positives and false negatives, respectively.

   Overall approach: the attacker embeds an if-else conditional prompt injection into submitted jailbreak references so that it activates only in the proliferation context and outputs attacker-prepared few-shot samples, thereby contaminating the classifier's training data; on this basis two types of attack are implemented—a false positive attack that induces shortcut learning, and an Omission Attack false negative backdoor based on concept removal.

   Problem setting and poisoning rate

   • The training set has 6k samples in total: 3k unsafe samples generated by proliferation + 3k WildChat safe queries (Section 2.1).
   • Each reference proliferates into about 3 training samples on average, so a 1% poisoning rate requires only 18 poisoned references; the authors note that the reported proportion is an upper bound because prompt injection does not always succeed (Appendix E).

   Conditional prompt injection delivery mechanism (Section 3.1)

   • IF branch (proliferation mode): upon detecting the proliferation context (e.g., "generate variants"-type instructions), it overrides the original task and samples uniformly from attacker-specified few-shot samples—benign queries for the utility attack, concept-removed queries for the safety attack.
   • ELSE branch (validation mode): when the defender validates whether a reference is a valid jailbreak, the original jailbreak payload executes normally and should be judged harmful by the validation model.
   • The authors argue that these trigger cues are inherent to the proliferation task and hard to remove without changing the synthetic data generation method. Appendices B and G give the full templates.

   False positive attack: inducing shortcut learning (Section 3.2)

   • Benign concepts carrying target features are injected into the training data via proliferation, causing the classifier to learn spurious associations between surface features (format, domain, entity names) and the unsafe label.
   • Four types of target features: format (plain text, MCQ, JSON, etc.), domain (e.g., law, math), entity names (e.g., company names), and the general user query distribution.

   Omission Attack: concept backdoor (Section 3.3)

   • The attacker selects a concept C (e.g., Harry Potter, a common n-gram), finds benign queries containing C, removes C with regex or an LLM, and then injects the modified queries as unsafe samples into training via prompt injection.
   • The authors' empirical observation: after training, the classifier associates the presence of C with safe; at inference, injecting C into jailbreaks (prepending the trigger once before a generally harmful query, and in jailbreak evaluation inserting it at every sentence boundary) causes them to be misclassified as safe even if these jailbreaks appeared in the same RR round.
   • The authors' hypothesis: C appears only in safe samples and never in structurally similar poisoned jailbreaks, and this asymmetry makes C a strongly discriminative safety feature.

   Training and decision settings (Section 4, Appendix A)

   • The target classifier is LlamaGuard 4 12B, fine-tuned with LoRA (rank 32, α=32); the proliferation models are Gemini 2.5 and 3 (Flash and Pro variants), and the authors say Gemini was chosen because its safety filters are configurable and it can perform the proliferation task, whereas GPT and Claude refuse.
   • Jailbreak references come from 300 each of PAIR, Cipher and Crescendo, and the attacker manipulates a small fraction of the PAIR ones; training stops early once validation-set accuracy (including all three attack methods) reaches at least 90% for each attack strategy.
   • Metrics: FPR for the utility attack, FNR for the safety attack; the harmfulness and goal consistency of poisoned references are judged by a two-stage LLM validation process (Appendix H).

   Collapse this answer

4. 04What experiments did the paper conduct?

   On LlamaGuard 4 12B, at a 1% poisoning rate, reaching up to 100% FPR and 96% FNR, and evaluating two types of defenses.

   Experimental Setup

   • Models under test: The target classifiers are LlamaGuard 4 12B (main experiments) and Prompt Guard 2 (86M) (generalization experiments in Figure 8); the proliferation models are Gemini 2.5/3 (Flash and Pro variants).
   • Data: Training set of 6k (3k proliferated unsafe + 3k WildChat safe samples); jailbreak references are 300 each from PAIR, Cipher, and Crescendo.
   • Baselines: No-poisoning RR (RR-CLEAN), untuned base model; the Omission Attack additionally has a direct poisoning (DP) baseline.
   • Metrics: FPR (utility attack), FNR (safety attack); positive samples are the held-out jailbreaks and harmful queries from Peng et al. (2024).
   • Evaluation set sizes: WildChat validation set n=1000, WildChat-4.8M held-out set n=1000, XSTest-SAFE n=250, LM-SYS-CHAT n=779, AdvBench 100 harmful queries, 150 each for the three jailbreak methods.
   • Validation: Two-stage LLM validation of poisoning references (Gemini-3-Flash-Preview): baseline PAIR has a Q1 harmfulness rate of 90.00% and a Q2 same-behavior rate of 89.00%; poisoned samples average 97.00% and 87.78% (Table 2).

   Main Results

   Format attack (1% poisoning, per Table 1, MMLU IID evaluation):

   Evaluation format	Poisoning target format	FPR
   MCQ	MCQ	100%
   JSON	JSON	100%
   HTML	HTML	100%
   YAML	YAML	100%
   Plain text	Plain text	90.51%

   • The authors state that format attacks barely transfer across formats (off-diagonal entries in Table 1 are mostly 0–23%), and that they generalize to OOD GPQA (e.g., MCQ 100%, JSON 98.21%); the clean baseline and base model have FPR of about 0% on all formats.
   • Domain attack (Figure 4): Under 1% poisoning, the average FPR across five target domains is 77.8% (Professional Law 98%, Econometrics 96%, International Law 75%, Machine Learning 73%, High School Biology 47%), and the average across non-target domains is 4.7%; the authors note spillover in Econometrics, possibly due to conceptual overlap.
   • Entity attack (Figure 5, Figure 26): The contrast-enhancement method reaches 98% FPR on "ChatGPT" queries and 18.8% on "Claude" queries (naive and hybrid methods are 47.5–70%); the authors state that constructing triggers from WildChat queries in the same distribution as training is key to stealthiness.

   Omission Attack (safety subversion)

   • n-gram concept deletion (Figure 7): With 1% poisoning, the Midjourney trigger yields 96% FNR on harmful queries; the Meshy trigger, unseen during training, transfers to 98% FNR. Direct poisoning (DP) yields higher FNR on jailbreak strategies (94–100%) but lower on general harmful queries (62–63%); the authors consider this attack especially effective against the specific attack strategies the RR must adapt to.
   • Rare 15-gram trigger (Figure 25; this concept appears only 8 times in the defender's WildChat distribution): With 1% poisoning, the short trigger yields 73% FNR on harmful queries and 22–66% on jailbreak strategies; the longer paraphrase trigger rises to 98% and 41–67%, with the short trigger at 0% on Cipher and the long trigger at 40.67%.
   • LLM semantic deletion (Harry Potter) (Figure 9): With 1% poisoning, 87% FNR on harmful queries and 55% FNR on HP-modified PAIR jailbreaks; the clean baseline is 6% and 0%, respectively. Naively concatenating HP character names yields only 14.7% FNR (clean baseline 5.3%), from which the authors conclude the backdoor depends on the model's understanding of the concept rather than a surface shortcut.

   Generalization and Mechanistic Analysis

   • Changing carrier, pathway, and classifier (Section 6.3, Figure 8): With Cipher as the carrier, 1% poisoning yields 91% FNR on harmful queries and 74% on PAIR, and the Meshy trigger transfers (89%/66%/18%/50% for harmful/PAIR/Crescendo/Cipher); DP can implant backdoors in both Llama Guard 4 and Prompt Guard 2 (86M), and the authors state that omission's generalization is not limited to RR.
   • Mechanistic analysis (Appendix F, Figure 20): Borrowing the refusal-direction method, on the poisoned Llama Guard 4, clean harmful prompts and harmful + random control strings maintain a positive (unsafe) projection in the upper layers, whereas harmful + trigger shifts toward the negative (safe) region starting around layer 30; the authors state this indicates omission shifts the internal safety representation rather than only perturbing the output head.

   Defense Evaluation (Section 7)

   • Input detection and sanitization: A guardrail LLM (Gemini 2.5 Flash Lite) is deployed following the PromptArmor implementation to check each reference; on 300 clean PAIR and 18 poisoned references per template (7 templates total), clean FPR is 3.8% and aggregated poisoned FNR is 10.3% (Figure 28); harder-to-detect templates such as SWE-bench have 24.1% FNR and MCQ format 18.5% FNR. Content removal after detection is also imperfect: 35% of true positives are over-deleted to empty strings, and 56% retain traces of template structure.
   • Model-level defense: When Meta SecAlign 70B is used as the proliferation model, the authors state it can ignore the injected template, and the FPR in the Professional Law target domain drops from 98% to 0%; however, the authors emphasize that its templates are statically hand-constructed, that stronger adaptive attackers may achieve higher bypass rates, and that the results should be viewed as a lower bound on attacker capability.

   Other Ablations and Analyses

   • Few-shot vs zero-shot (Appendix D.1, Figure 21): 1% few-shot poisoning yields 94–100% FPR and nearly saturates all MMLU domains; zero-shot does not saturate even at 5% poisoning, with many domains below 60–70%.
   • Non-prompt-injection variant (Appendix D.4): Without prompt injection, the ChatGPT/Claude/Gemini noun attacks achieve ASR of 9.5%, 6%, and 8.5%, respectively, under 1% poisoning.
   • SWE-Bench Verified targeting (Section 5.4): Reaches 100% FPR on coding prompts but does not transfer to general WildChat queries (0% FPR).
   • General distribution attack (Figure 6): 5% poisoning yields 39–50% FPR across four evaluation sets, versus 0.8–3.2% for the clean baseline; greedy set-cover is used to select 3-grams to maintain effectiveness at low poisoning rates.
   • Entity attack with batch size 8 (Figure 27): 98.18% ASR for the Gemini target, but collateral effects of 47.20% on Claude and 56.36% on ChatGPT; the authors state that a smaller batch yields a higher success rate but a broader scope of impact.

   Collapse this answer

5. 05What are some points that can be further explored?

   The authors note limitations such as the attack depending on prompt injection succeeding and the evaluation being limited to a single classifier and proliferation model.

   Limitations and future directions noted by the authors

   • Prompt injection success "gap" (Section 8): the strongest false positive attacks depend on prompt injection succeeding during the proliferation stage; the authors believe that progress in prompt injection defenses may reduce attack effectiveness, but guaranteeing robustness to prompt injection remains an open problem; dedicated rewriting models may be less susceptible to injection but may be worse proliferation models.
   • Non-prompt-injection variants are weaker (Section 8, Appendix D.4): non-prompt-injection variants were tested in the targeted noun setting, and the success rate was lower as expected.
   • Evaluation scope (Section 8): the experiments used only LlamaGuard 4 as the target classifier and Gemini as the proliferation model; the authors state that different architectures may exhibit different vulnerability profiles.
   • Lower-bound nature of the static template (Section 7): the current prompt injection attack uses a static handcrafted template; the authors believe that stronger adaptive attackers may achieve higher bypass rates, and downstream consequences will become more prominent as prompt injection methods strengthen.
   • Defense trade-offs (Appendix D.6): the authors discuss three directions for mitigation (prompt injection defenses for the proliferation model, harmfulness filtering before and after proliferation, benign data augmentation), and point out their respective trade-offs, e.g., benign data may need to dominate the training mix by orders of magnitude, which may limit adaptation speed.

   Experimental coverage

   • The target classifiers tested are LlamaGuard 4 12B and Prompt Guard 2 (86M), the latter appearing only in the generalization experiment in Figure 8.
   • Defense evaluation includes two types: PromptArmor-style input detection sanitization (300 clean + 18 poisoned references per template, 7 templates in total) and Meta SecAlign 70B as the proliferation model (only in the Professional Law targeted false positive setting).
   • The attacker-manipulated references come from PAIR (a small subset of the 300); jailbreak evaluation covers 150 each of PAIR, Cipher, and Crescendo, and general harmful queries are 100 from AdvBench.
   • Poisoning rate experiments cover 1%–16.7% (format attacks, Figure 21) and 1%–5% (general distribution attacks, Figure 6); the main results focus on 1%.
   • The paper does not report a specific value for the prompt injection success rate π (Appendix E only gives a formalization and states that the actual poisoning rate is below the reported upper bound), nor does it report the number of repetitions or variance for each experiment.

   Collapse this answer

6. 06Summarize the main content of the paper

   Practical poisoning attacks against the RR framework: at a 1% poisoning rate, achieving targeted false positives and concept-backdoor false negatives, and evaluating two types of defenses.

   An attack paper targeting a deployed safety mechanism: the authors study how the proliferation process of the Rapid Response (RR) jailbreak detection framework can be poisoned, where attackers can only submit and modify unsafe-class jailbreak references (keeping them harmful and the target unchanged), and cannot modify benign data or labels.

   • Method: Using if-else conditional prompt injection to make poisoned references output attacker-prepared few-shot samples during proliferation and preserve the original jailbreak during validation; based on this, two types of attacks are implemented—targeted false positive attacks via shortcut learning (format/domain/entity/general distribution), and the Omission Attack: removing concept C from unsafe samples so that the classifier associates the appearance of C with the safe label, and injecting C into a jailbreak at inference time causes a false negative.
   • Utility disruption results (LlamaGuard 4 12B, 1% poisoning): format attacks reach up to 100% FPR on the target format (MCQ, JSON, Table 1); domain attacks reach 98% FPR on Professional Law and an average of 77.8% across five target domains (Figure 4); entity attacks reach 98% FPR on "ChatGPT" queries and 18.8% on "Claude" (Figure 26).
   • Safety disruption results: at 1% poisoning, the Midjourney n-gram trigger reaches 96% FNR on general harmful queries, and the unseen Meshy trigger transfers at 98% (Figure 7); the Harry Potter semantic trigger reaches 87% FNR on harmful queries and 55% on HP-modified PAIR jailbreaks (Figure 9).
   • Defense evaluation: PromptArmor-style input detection aggregates to 10.3% FNR against poisoned references (Figure 28); using Meta SecAlign 70B as the proliferation model reduces the target-domain FPR from 98% to 0%, but the authors emphasize that the static template is only a lower bound on attacker capability.
   • Authors' conclusion: there is a structural tension between RR's rapid adaptation and robustness—proliferation amplifies a single poisoned input into a large number of training samples; the authors argue that new safety mechanisms themselves need to be hardened before deployment, and disclosed to potentially affected parties before publication.

   Collapse this answer

Related papers