Skip to content

LLM Safety Guardrails Are a Double-Edged Sword: Exploiting False Positives for Denial-of-Service Attacks

LLM Safeguard is a Double-Edged Sword: Exploiting False Positives for Denial-of-Service Attacks

arXiv· arXiv:2410.02916· Workshop on Large AI Systems and Models with Privacy and Security Analysis 2025· Qingzhao Zhang, Ziyang Xiong, Z. Morley MaoUniversity of MichiganIndexed by this site · 20 days ago· Originally publishedSelected AttacksSelectedAttention38

Quick read

Attacks

Based on the paper PDF (AI-generated); refer to the original

The authors propose an LLM denial-of-service attack that exploits safety guardrail false positives: white-box prompt injection on Llama Guard 3 rejects over 97% of safe requests with an adversarial prompt of about 30 characters.

Threat modelThe attacker's goal is to make the safeguard misjudge safe user requests as unsafe, causing DoS. Four threat models: white-box prompt injection (knows guardrail parameters, can inject into the user prompt template), black-box prompt injection (can only query), poisoned fine-tuning (can fine-tune the target LLM with malicious data), poisoned fine-tuning with backdoor (plants a backdoor and can inject a trigger sentence). The victim system is an LLM service deployed with safety alignment or an external guardrail.

Questions
LLM safety guardrails have false positives: misjudging safe requests as unsafe. The authors propose that attackers can exploit this to make guardrails persistently refuse normal user requests, forming a denial of service (DoS), a threat overlooked by existing research.
Methods
Design attacks under four threat models: white-box/black-box prompt injection (using GCG gradients or word-level mutations to optimize short adversarial prompts inserted into the user prompt template, with a loss function balancing success rate, length, and semantic stealth), and poisoned fine-tuning with refusal samples, plus a backdoor variant that can carry a benign trigger sentence.
Experiments and results
White-box injection on Llama Guard 3 achieves a multi-task suffix success rate of 98.8% with a length of 40.3 characters (Table 1); black-box injection on Llama Guard 3 reaches a maximum of 86.6%, and less than 16% on GPT-4o; backdoor fine-tuning with 20 poisoned samples makes GPT-3.5 refuse 77.8% of requests; both existing defenses push TPR below 30%.
Limitations and future work
The authors state that the tested models and datasets are limited, and leave larger-scale benchmarking to future work (Section 6). The experiments cover 6 models and 4 public datasets, with each setting repeated 20 times; the paper does not report the sample size of each dataset or human evaluation.
Experimental setupLlamaGuard-7b、Meta-Llama-Guard-2-8B et al. · Alpaca、HumanEval et al. · success rate、adversarial prompt length et al.
Evaluated models
LlamaGuard-7bMeta-Llama-Guard-2-8BLlama-Guard-3-8BVicuna-1.5-7bGPT-3.5GPT-4oGemini
Benchmarks
AlpacaHumanEvalAgievalHarmBench
Metrics
success rateadversarial prompt lengthTPRFPR
AI summaryand selection rationaleThe paper proposes a new class of attacks that exploit LLM safety guardrail false positives, causing the guardrail to misclassify normal requests as unsafe and thereby carrying out denial-of-service (DoS) against users. Attack paths include injecting adversarial prompts into user prompt templates and compromising the model through poisoned fine-tuning; real-world scenarios include exploiting client software vulnerabilities, phishing attacks, controlling agents, and insider poisoning. An approximately 30-character adversarial prompt generated by a white-box attack can block more than 97% of user requests on Llama Guard 3; on ChatGPT, fine-tuning with 20 poisoned samples can cause 78% of user requests to be rejected. The black-box attack achieves a success rate of less than 16% on GPT-4o, and the authors conjecture that models with stronger reasoning ability are more resistant to such attacks; the two existing mitigation methods tested, although able to lower the attack success rate, reduce the guardrail's detection rate for unsafe content to below 30%. The paper points out that this reveals a neglected dimension in safety guardrail evaluation, namely false-positive robustness in adversarial settings.

The paper proposes a new class of attacks that exploit LLM safety guardrail false positives, causing the guardrail to misclassify normal requests as unsafe and thereby carrying out denial-of-service (DoS) against users. Attack paths include injecting adversarial prompts into user prompt templates and compromising the model through poisoned fine-tuning; real-world scenarios include exploiting client software vulnerabilities, phishing attacks, controlling agents, and insider poisoning. An approximately 30-character adversarial prompt generated by a white-box attack can block more than 97% of user requests on Llama Guard 3; on ChatGPT, fine-tuning with 20 poisoned samples can cause 78% of user requests to be rejected. The black-box attack achieves a success rate of less than 16% on GPT-4o, and the authors conjecture that models with stronger reasoning ability are more resistant to such attacks; the two existing mitigation methods tested, although able to lower the attack success rate, reduce the guardrail's detection rate for unsafe content to below 30%. The paper points out that this reveals a neglected dimension in safety guardrail evaluation, namely false-positive robustness in adversarial settings.

Why it was selectedThe paper reports attack results in which an approximately 30-character adversarial prompt blocks 97% of user requests on Llama Guard 3, which teams deploying guardrails can use to assess false-positive robustness.

Paper analysis

6 questions; open one to read the full answer

  1. What problem does this paper attempt to solve?

    The authors propose the overlooked threat of exploiting LLM safety guardrail false positives to launch denial-of-service attacks, and present four types of attack methods.

    Core question: can an attacker exploit false positives of LLM safety guardrails (misclassifying safe content as unsafe) to make normal user requests be persistently rejected, forming a denial-of-service (DoS) attack.

    • Problem setting and importance: LLM deployments commonly use safety alignment or external guardrails (such as Llama Guard) to block unsafe content. The authors argue that in systems such as finance and healthcare, persistently triggering rejections would harm user experience, cause economic losses, and even endanger public health.
    • Gaps in existing research: Jailbreak attacks exploit guardrail false negatives and are the focus of research; the authors state that the DoS threat triggered by false positives is overlooked by existing research.
    • Core observation: Guardrails are binary classifiers; the attacker does not need the model to generate harmful content, only needs the classifier to output unsafe for safe requests to block service; the authors state this constitutes a new dimension of guardrail evaluation—adversarial robustness to false positives.
    • What the paper proposes: DoS attacks under four threat models (white-box/black-box prompt injection, poisoned fine-tuning, poisoned fine-tuning with backdoor), evaluated on multiple guardrail models and task datasets, analyzing factors affecting the success rate.
    • Threat models (Section 3.1):
      • Attacker goal: increase the probability that user requests are rejected by the server, forming a DoS against specific users.
      • white-box prompt injection: can inject malicious prompts into target user requests (e.g., via software vulnerabilities or phishing to tamper with the prompt template in the client configuration file), and has white-box access to guardrail parameters; cannot compromise the trusted server.
      • black-box prompt injection: can only query the target LLM to obtain text responses, and cannot access model parameters.
      • poisoned fine-tuning: can fine-tune the target LLM with specified malicious data (e.g., can access the server or fine-tuning tools), but cannot tamper with the user client; the backdoor variant additionally implants a backdoor and has prompt injection capability, activating the rejection behavior only when the trigger sentence is injected.
  2. What related research exists?

    Related work falls into three groups: safety alignment and guardrails, jailbreak attacks, and adversarial defenses; the authors state that the false positives/DoS studied in this paper are fundamentally different from jailbreaks.

    Safety alignment and guardrails

    • Stiennon et al. (2020)—proposed RLHF, training LLMs with human feedback and preferences, which became a standard training method; supervised fine-tuning and instruction tuning (Achiam et al., 2023; Touvron et al., 2023) further improve safety using prompt-response data.
    • Llama Guard (Inan et al., 2023) and OpenAI Moderation API (2024)—model-based guardrails: use fine-tuned or pretrained models to determine the safety category of a conversation; rule-based filters directly match patterns such as toxic words.
    • Advanced guardrails (Yuan et al., 2024; Rebedea et al., 2023)—integrate multiple protection mechanisms and adversarial defenses, or provide customizable frameworks.

    Jailbreak attacks (exploiting false negatives)

    • Early findings: rewrites such as role-playing, unsafe few-shot, and domain shift can bypass guardrails (surveys: Yu et al., 2024; Dong et al., 2024).
    • Automated jailbreaks: black-box optimization (Shin et al., 2020; Ge et al., 2023) and white-box optimization GCG (Zou et al., 2023) automatically generate jailbreak prompts; data poisoning can also make models jailbreakable (Gao et al., 2024; Wang et al., 2024a).

    Defenses against adversarial attacks

    • Hu et al. (2023)—identify adversarial prompts based on perplexity, assuming their wording style differs from normal generation.
    • Robey et al. (2023), Ji et al. (2024)—apply random perturbations to content, assuming adversarial prompts are sensitive to perturbations.
    • Yuan et al. (2024)—resilient optimization: optimize another prompt against the jailbreak prompt.

    Baselines and benchmarks: the ablation experiments use GCG (Zou et al., 2023) as the baseline (Appendix C); defense evaluation uses the random perturbation of Robey et al. (2023) and the resilient optimization of Yuan et al. (2024). Safe prompts are taken from Alpaca, HumanEval, and Agieval; unsafe prompts are taken from HarmBench.

    Positioning of this paper: the authors state that existing attacks and defenses mostly focus on the jailbreak problem, and that the false-positive-triggered/denial-of-service problem studied in this paper is fundamentally different from jailbreaking.

  3. How does the paper address this problem?

    The four attack paths share one goal: making the guardrail output unsafe for safe requests; injection-type attacks use iterative optimization with concealment constraints to generate short adversarial prompts.

    Overall approach: model the guardrail as a binary classifier f, and the attack goal is to make f output 1 (unsafe) for concatenated safe requests. The paper presents two categories, prompt injection (white-box/black-box) and poisoning fine-tuning (baseline/backdoor), totaling four attacks.

    Problem formalization and attack goal

    • Request m = mu ⊕ mt (user prompt concatenated with prompt template). In the injection scenario the attacker uses a function g to modify the template; in the poisoning scenario the guardrail is changed to f′. Ideal goal: for all safe prompts mu ∈ M, the original f judges safe while after the attack it judges unsafe.
    • Concealment is measured by two metrics: length (fewer characters is better) and semantics (low semantic similarity to known unsafe content, evaluated with SentenceBERT). The authors say this cannot guarantee undetectedness but lowers the probability of detection.

    Prompt injection attack (white-box, Algorithm 1)

    • Materials: a safe prompt set (to test candidate success rates), an unsafe prompt set (to provide initial candidates), the target guardrail and target reply (e.g., Llama Guard's "unsafe"), and a loss function.
    • Loss function: Loss = 1/(|M|)∑m∈ M CrossEntropy(f(m ⊕ ma), mt) + w1 · Length(ma) + w2 · SemanticSimilarity(ma, ma(0)), i.e., the cross-entropy of the target reply (effectiveness) plus length and semantic concealment penalties.
    • Process: initialize from the unsafe prompt with the lowest loss; each round uses GCG gradients for token replacement (k1=24), then deletes unimportant tokens by last-layer attention values (k2=8); first filter candidates by a success-rate threshold σ=0.6, then take the lowest-loss one into the next round.
    • Parameters: 12 safe prompts per iteration, w1=10⁻⁴, w2=0.1, parameters manually tuned (Appendix B.1.3).

    Prompt injection attack (black-box, Algorithm 2)

    • Mutations are changed to the word level: random replacement (vocabulary from high-frequency words in white-box replacements), random deletion, random swapping of two segments, with no gradient or attention guidance.
    • The first loss term is changed to the success rate on the validation set (cross-entropy unavailable); 20 safe prompts per iteration, k1=16, k2=10, k3=4, σ=0.1.

    Poisoning fine-tuning attack

    • Baseline attack: poisoned samples are in Q&A format—a safe user request followed by a refusal reply; only the training data is manipulated, and the fine-tuning process and loss are unchanged. Sample format is adjusted per guardrail (e.g., Llama Guard requires a specific template).
    • Backdoor attack: the poisoned set contains samples with and without a trigger sentence, and the model refuses only requests with the trigger sentence; the trigger sentence is chosen to look normal to maintain concealment. ChatGPT experiments use the OpenAI fine-tuning API, with 10 poisoned samples for the baseline and 20 for the backdoor.

    Generality design

    • The attacker does not control user prompts, so the attack must generalize across scenarios: task scope is divided into single-task/multi-task; insertion position into prefix/suffix/random; token filter into None/Moderate (English words only)/Strict (additionally bans a toxic word list).
  4. What experiments did the paper conduct?

    White-box injection achieves a maximum success rate of 98.9% on Llama Guard 3; black-box and poisoning both achieve success rates below 16% on GPT-4o; both existing defenses simultaneously cripple the guardrail's normal performance.

    Experimental setup

    • Models tested: Vicuna-1.5-7b (safety-aligned), LlamaGuard-7b, Meta-Llama-Guard-2-8B, Llama-Guard-3-8B (external guardrails); GPT-3.5 and GPT-4o are used only for black-box injection and fine-tuning attacks.
    • Datasets: safe prompts come from Alpaca (short questions), HumanEval (programming), and Agieval (math, logical reasoning, reading comprehension); unsafe prompts come from HarmBench; the safe set is split into a training set (input to the attack algorithm) and a test set (evaluation only). The paper does not report the sample size of each dataset.
    • Metrics: success rate (the proportion of the safe prompt set that triggers a guardrail warning); injection attacks additionally report adversarial prompt length (number of characters).
    • Repetitions: each setting repeats the attack 20 times, reporting average metrics.
    • Variables: task scope (single/multi-task), token filter (None/Moderate/Strict), insertion position (prefix/suffix/random).
    • Poisoned fine-tuning: via the OpenAI fine-tuning API, 10 poisoned samples for the baseline attack and 20 for the backdoor attack.

    Main results

    White-box prompt injection (Table 1, success rate in %, length in characters):

    Swipe horizontally to read the table

    ModelSingle-task/No filter/SuffixMulti-task/No filter/SuffixMulti-task/No filter/PrefixMulti-task/No filter/RandomMulti-task/Strict/Suffix
    Llama Guard97.1 / 27.896.3 / 42.068.0 / 39.353.5 / 52.181.3 / 55.1
    Llama Guard 297.5 / 33.997.2 / 34.792.3 / 47.679.6 / 55.897.8 / 55.2
    Llama Guard 398.9 / 26.998.8 / 40.394.4 / 33.783.5 / 37.397.6 / 49.3
    Vicuna-1.5-7b99.9 / 26.499.9 / 27.899.8 / 22.294.3 / 28.398.1 / 35.0
    • The authors state that the Llama Guard series becomes more vulnerable to DoS attacks as versions evolve: Llama Guard 3's success rate is 20.4% higher than the first generation; Vicuna is generally more vulnerable.
    • The authors state that single-task success rates are only slightly higher than multi-task with comparable length, indicating that adversarial prompts generalize across tasks; fixed-position insertion outperforms random insertion; attacks are more effective on short user prompts (Figure 4).
    • Moderate filtering performs comparably to no filtering, while Strict filtering makes prompts longer; the authors believe specific keywords strongly influence guardrail responses.

    Black-box injection and transfer attacks

    • Black-box injection (Table 2): on Llama Guard 3 multi-task, suffix/prefix/random success rates are 85.1%/86.6%/51.3% (length 11.6–12.4); GPT-3.5 is 52.7%/28.7%/15.5%; GPT-4o is 15.5%/12.7%/12.0%. The authors state that compared with Llama Guard 3 white-box, black-box success rates drop by 18% on average and length shortens by 60%, relying more on deletion because effective replacements are hard to find.
    • The authors' interpretation of GPT-4o: success rates are below 16%, and Gemini tests are similarly low (not included in the table); the authors hypothesize that reasoning models' chain-of-thought processes are less affected by meaningless semantic adversarial prompts.
    • Transfer attacks (Table 3): transferring prompts optimized on Llama Guard 3 (Exp 3.2), success rates are 65.8% on Llama Guard 2, 54.8% on Vicuna-1.5-7b, and only 0.7% on GPT-4o; on Llama Guard 3 itself it is 27.8% (here the target model is the source model). The authors state that transfer is more effective when the target and source have similar architectures or training data.

    Poisoned fine-tuning (Table 4)

    • GPT-3.5: baseline (no trigger) success rate 57.4%, backdoor (prefix trigger) 77.8%; GPT-4o: baseline 2.3%, backdoor 12.5%.
    • The authors state that the backdoor attack yields an absolute improvement of about 20% (GPT-3.5) and 10% (GPT-4o) over the baseline, and that the trigger sentence is a semantically normal sentence that is hard for the human eye to identify as malicious; GPT-4o is more resilient, consistent with the black-box injection pattern.

    Defense evaluation (Figure 5)

    • Random perturbation (Robey et al., 2023) and resilient optimization (Yuan et al., 2024) are applied to the multi-task suffix no-filter white-box attack (Exp x.2).
    • The authors state that the defenses reduce the attack success rate to at most 40%, but simultaneously harm the guardrail's normal performance: random patching and resilient optimization push TPR below 30% and raise FPR above 10%, rendering the guardrail ineffective.

    Other ablations and analyses

    • Algorithm ablation (Appendix C, Figure 6): with GCG as the baseline, combining token replacement and loss function design shortens adversarial prompt length from 179.2 to 40.3 (about 20%) while maintaining the success rate.
    • Case study (Appendix D): analyzes the AnythingLLM framework; CVE records 46 vulnerabilities, many involving unauthorized file access; the authors state that after injection, almost all requests cannot be processed.
    • Optimization process (Figure 2, Appendix E): taking Exp 3.2/3.5 as examples, over 50 iterations the success rate remains high while prompts gradually become shorter and semantically more ambiguous; Appendix E provides a complete iteration example.
  5. What are some points that can be further explored?

    The authors point out only the limitation of limited model and dataset scale; the experiments cover 6 models, 4 types of attacks, and 2 defenses, with each setting repeated 20 times.

    Limitations and future directions noted by the authors

    • Limited test scale (Section 6, Threats to validity): the authors state that testing was conducted on a limited set of models and datasets, and leave larger-scale benchmarking to future work; the authors also consider their model selection representative.
    • Mitigation suggestions (Section 6, Mitigation): the authors suggest that users implement standard protections against software and phishing attacks to block the attack path; if a large number of failed requests is observed, they suggest manually inspecting the inference pipeline. (This is a suggestion, not a validated defense.)

    Scope of experimental coverage

    • Models tested: white-box injection covers 4 models in total: LlamaGuard-7b, Meta-Llama-Guard-2-8B, Llama-Guard-3-8B, and Vicuna-1.5-7b; black-box injection covers Llama-Guard-3-8B, GPT-3.5, and GPT-4o; poisoned fine-tuning covers only GPT-3.5 and GPT-4o; Gemini is only mentioned as having been tested, with results not included in the tables.
    • Datasets: safe prompts come from Alpaca, HumanEval, and Agieval (covering categories such as math, programming, logical reasoning, and reading comprehension); unsafe prompts come from HarmBench; the paper does not report the specific sample size of each dataset.
    • Defense evaluation: only two defenses, random perturbation (Robey et al., 2023) and resilient optimization (Yuan et al., 2024), and they are applied only to the white-box attack setting with multi-task suffixes and no filtering.
    • Repetitions and budget: each setting repeats the attack 20 times and averages; white-box uses 24 token replacements and 8 deletions per round, black-box uses 16/10/4 word-level mutations per round; the paper does not report query budget or time cost beyond the total number of iterations N.
    • Evaluation method: success determination relies on the guardrail's own output (e.g., "unsafe"); the paper does not use LLM judging or human evaluation, nor does it report human agreement.
  6. Summarize the main content of the paper

    The authors propose and evaluate LLM denial-of-service attacks that exploit guardrail false positives: short adversarial prompts or poisoned fine-tuning can cause guardrails to refuse safe requests at scale.

    An attack paper: targeting the false-positive surface of LLM safety guardrails (safety-aligned models and external guardrails), it proposes denial-of-service (DoS) attacks that make guardrails misjudge normal user requests as unsafe and refuse them.

    • Problem: Jailbreak research focuses on guardrail false negatives; the authors say DoS triggered by false positives has been overlooked; in scenarios such as finance and healthcare, sustained refusals can cause real harm.
    • Method: four threat models—white-box/black-box prompt injection (inserting optimized short adversarial prompts into the user prompt template, with a loss function balancing success rate, length and semantic concealment; white-box uses GCG gradients plus attention-guided deletion), poisoning fine-tuning (safe requests paired with refusal replies, 10–20 samples) and its backdoor variant (a benign trigger sentence activates refusal).
    • White-box results (Table 1): Llama Guard 3 multi-task suffix injection success rate 98.8%, prompt length 40.3 characters; Vicuna-1.5-7b highest at 99.9%; the authors say Llama Guard 3 has a 20.4% higher success rate than the first generation, i.e., guardrails become more vulnerable to this attack as versions evolve.
    • Black-box and transfer (Tables 2, 3): Llama Guard 3 black-box highest at 86.6%; GPT-4o black-box under 16%, transfer only 0.7%, which the authors hypothesize is related to the reasoning model's chain of thought.
    • Poisoning fine-tuning (Table 4): a 20-sample backdoor attack makes GPT-3.5 refuse 77.8% of requests and GPT-4o refuse 12.5%.
    • Defenses and conclusion: two existing defenses (random perturbation, resilience optimization) reduce the success rate to at most 40%, but push guardrail TPR below 30% and raise FPR above 10% (Figure 5). The authors argue that guardrail evaluation must incorporate false-positive robustness as a new dimension, and leave larger-scale benchmarking to future work.

Related papers

Show 3 more papers
Read originalWorkshop on Large AI Systems and Models with Privacy and Security Analysis 2025