Skip to content

Adversarial Prompt Evaluation: A systematic benchmark evaluation of guardrails against prompt input attacks on large language models

Adversarial Prompt Evaluation: Systematic Benchmarking of Guardrails Against Prompt Input Attacks on LLMs

arXiv· arXiv:2502.15427· Giulio Zizzo, Giandomenico Cornacchia, Kieran Fraser, Muhammad Zaid Hameed, Ambrish Rawat, Beat Buesser, Mark Purcell, Pin-Yu Chen, Prasanna Sattigeri, Kush VarshneyIBM ResearchIndexed by this site · 23 days ago· Originally publishedEvaluation

Quick read

Evaluation and benchmarks

Based on the paper PDF (AI-generated); refer to the original

The authors conducted a systematic benchmark evaluation of 15 guardrails on 27 malicious and benign datasets, and the authors state that the OOD performance of a simple classifier baseline can compete with various existing guardrails.

Questions
Jailbreak attack styles are continually increasing, and external guardrails are widely used to protect LLMs, but the evaluation datasets, protocols, and metrics of different defenses vary greatly, there is a lack of systematic benchmarks for guardrails, and performance under OOD attacks is unclear.
Methods
The authors pool 27 malicious and benign datasets and split them into in-distribution and OOD parts, uniformly evaluating 15 guardrails (two types: detection-based classifiers and LLM-as-judge), with metrics including AUC, ACC, F1, Recall, and Precision, and recording memory, latency, and throughput.
Experiments and results
The authors report that guardrail performance varies greatly with jailbreak style; self-trained BERT, DeBERTa, and GPT2 classifiers perform comparably to more expensive open-source and closed-source guardrails on OOD data; LLM-based guardrails have higher detection rates but also higher false positive rates; SmoothLLM has relatively many false positives on BoolQ and XSTest.
Limitations and future work
Main limitations noted by the authors: because some defenses are computationally expensive and the number of datasets is large, the final evaluation data were subsampled (Section 7). The evaluation covers 15 guardrails and 27 datasets; the in-distribution evaluation set has 11387 entries, and OOD has 10327 benign plus 376 malicious; the paper does not report statistical significance tests.
Experimental setupRandom Forest、BERT et al. · AART、AttaQ et al. · AUC、ACC et al.
Evaluated models
Random ForestBERTDeBERTaGPT2Protect AI (v2)Llama-Guard 2Langkit Injection DetectionSmoothLLMPerplexityOpenAI Moderation (text-moderation-007)Azure AI Content SafetyNemo Inspired Input RailLangkit Proactive DefenceGranite Guardian 3.0
Benchmarks
AARTAttaQAutoDANDo-Not-AnswerGandalf Ignore InstructionsGCGHarmful BehaviorsJailbreak PromptsPrompt ExtractionSAPChatGPT DANJailbreakchatMalicious InstructXSTestToxicChatAlpacaAwesome ChatGPT PromptsBoolQNo RobotsPuffinSuper Natural InstructionsUltraChatDollyHuman Preferenceinstruction-datasetOrca DPO PairsPIQA
Metrics
AUCACCF1RecallPrecisionTP rateFP rateMemoryLatencyThroughput
AI summaryThe researchers conducted a systematic benchmark evaluation of 15 guardrails used to defend against jailbreak attacks on large language models, covering a large number of malicious and benign datasets. The paper points out that because the range of jailbreak types used during alignment is narrow, not all defenses can handle new out-of-distribution attacks, and the lack of systematic organization of defenses also creates a clear gap in practical applications. The evaluation finds that defense performance varies significantly with the jailbreak style faced; based on existing available evaluation datasets, simple baselines can compete with many state-of-the-art defenses on out-of-distribution attacks. The code is released in the IBM/Adversarial-Prompt-Evaluation repository on GitHub.

The researchers conducted a systematic benchmark evaluation of 15 guardrails used to defend against jailbreak attacks on large language models, covering a large number of malicious and benign datasets. The paper points out that because the range of jailbreak types used during alignment is narrow, not all defenses can handle new out-of-distribution attacks, and the lack of systematic organization of defenses also creates a clear gap in practical applications. The evaluation finds that defense performance varies significantly with the jailbreak style faced; based on existing available evaluation datasets, simple baselines can compete with many state-of-the-art defenses on out-of-distribution attacks. The code is released in the IBM/Adversarial-Prompt-Evaluation repository on GitHub.

Paper analysis

6 questions; open one to read the full answer

  1. What problem does this paper attempt to solve?

    Existing guardrail evaluations are scattered and data coverage is narrow; the authors systematically benchmark 15 guardrails on a variety of jailbreak and benign data.

    Core problem: there is a lack of systematic benchmarks for jailbreak guardrails, and the performance of existing defenses under out-of-distribution (OOD) attacks is unclear.

    • Setting and importance: LLMs can be manipulated by jailbreak and prompt injection attacks to produce unsafe behavior, and external guardrails are needed beyond alignment training; the authors believe that jailbreak styles keep growing, and defenses aligned on narrow-distribution data struggle to cope with new OOD attacks.
    • Existing gaps: the authors point out that in the literature, the evaluation methods, representative datasets and metrics of various defenses differ greatly (such as string matching or BERT classification), and there is a lack of benchmark suites specifically for guardrails, creating a gap in practical application.
    • Core observation: the authors hypothesize that guardrail performance depends on the jailbreak style faced, and that based on currently available evaluation data, simple baselines may be comparable to the OOD performance of various state-of-the-art guardrails.
    • Paper output: a benchmark evaluation framework covering 27 malicious and benign datasets and 15 guardrails, uniformly using multiple classification metrics, and additionally comparing memory, inference time and scalability, ultimately giving deployment-oriented guardrail selection recommendations.
    • Research questions: RQ1 whether existing benchmark datasets are sufficient to evaluate guardrail quality; RQ2 how guardrails compare under memory, inference time and scalability constraints; RQ3 how to recommend guardrails to practitioners.
  2. What related research exists?

    Related work is divided into three groups: attacks, defenses and benchmarks; the authors state that integration work already exists for attacks and benchmarks, but guardrails lack dedicated benchmarks.

    Attack Methods

    • GCG (Zou et al., 2023) and AutoDAN (Zhu et al., 2023 / Liu et al., 2023): optimization-based jailbreak attacks; this paper uses their datasets as sources of malicious samples.
    • Indirect prompt injection (Abdelnabi et al., 2023; Liu et al., 2023): attacks carried out via indirect injection; the authors classify attacks by generation method into optimization-based, manual, and parameter-based exploiting sampling decoding strategies.

    Defense Methods

    • Perplexity filtering and rewriting (Jain et al., 2023): uses perplexity to identify adversarial perturbations; this paper reproduces its perplexity filter.
    • SmoothLLM (Robey et al., 2023): defends against GCG-type attacks by perturbing prompts; the authors note its high computational cost and that it targets only specific jailbreak styles, and that the original paper did not expose its low performance on benign datasets.
    • NeMo Guardrails (Rebedea et al., 2023), Llama-Guard (Inan et al., 2023), LangKit (2024): programmable guardrails, LLM safety classifiers, and similarity/proactive defense tools, all included in this paper's evaluation.

    Benchmarks and Datasets

    • HarmBench (Mazeika et al., 2024), Jailbreakbench (Chao et al., 2024), EasyJailbreak (Zhou et al., 2024): integrate jailbreak attacks for easy comparison; the authors note that defenses have not received equal attention and that there is currently no benchmark suite specifically for guardrails.
    • safetyprompts.com (Röttger et al., 2024): a repository aggregating safety evaluation datasets.

    Baselines and Benchmarks: this paper's baseline methods include Random Forest (unigram features), BERT, DeBERTa, GPT2 classifiers, and the Vicuna-13b refusal rate baseline; the evaluation uses 27 datasets (Table 1), including AdvBench-family GCG/Harmful Behaviors, XSTest, ToxicChat, etc.

    Positioning: the authors state that existing work mostly integrates the attack side, whereas this paper specifically conducts a systematic benchmark evaluation for guardrails and examines whether model complexity brings better OOD generalization.

  3. How does the paper address this problem?

    27 dataset splits are pooled into ID/OOD, 15 detection-based and LLM-based guardrails are evaluated with unified metrics, and resource overhead is also measured.

    Overall approach: build a unified guardrail benchmark evaluation framework that compares two types of defenses under the same dataset splits and metrics — detection-based (classifiers/filters external to the LLM) and LLM-as-judge-based (using an LLM to judge and filter malicious prompts).

    Datasets and Splits

    • Dataset compilation: 27 datasets, divided by target type into jailbreak, benign, and mixtures of both (XSTest, ToxicChat); prompt types cover instruction, question, artificial attack, role playing, harmful/toxic behavior, chat (Table 1).
    • ID/OOD split: the 18 in-distribution datasets are split 80% training, 20% test, and the training set is further split into 20% validation; the 9 OOD datasets do not participate in training or validation of any self-trained model. This split applies only to the authors' own trained models (Random Forest and Transformer classifiers).
    • Evaluation set construction: from each in-distribution test split, at most 2000 entries are randomly sampled per set and deduplicated, yielding 11387 entries (9543 benign, 1844 malicious); the OOD portion is 10327 benign plus 376 malicious. Duplicate samples within and across datasets are removed.

    Detection-Based Defenses

    • Perplexity Threshold: reproduces the perplexity filter of Jain et al., computing perplexity with GPT-2, with the threshold set to the maximum perplexity over all AdvBench prompts.
    • Random Forest: trained on unigram features (lowercased, tokenized by words and punctuation).
    • Transformer classifiers: BERT, DeBERTa, GPT2 architectures, fine-tuned on the training set for malicious/benign binary classification.
    • Off-the-shelf detectors: LangKit Injection Detection (cosine similarity with known jailbreak embeddings), ProtectAI v2 (deberta-v3-base fine-tuned, 86M parameters), Azure AI Content Safety (jailbreak endpoint, version 2023-10-01-preview), OpenAI Moderation (text-moderation-007, taking the highest probability among harmful categories).

    LLM-as-Judge-Based Defenses

    • Vicuna baseline: uses whether Vicuna-13b refuses as the decision, automatically analyzing outputs according to the refusal keyword list of Robey et al. (keyword list in Appendix A.2).
    • SmoothLLM: perturbs prompts multiple times to disrupt adversarial payloads; the authors note its high computational overhead.
    • LangKit Proactive Defence: requires the LLM to repeat a secret string; when attacked, the model does not repeat it and instead responds to the attack prompt.
    • NeMo Inspired Input Rail: inspired by the NeMo input moderation rail, adds extra instructions and splits the template after the system prompt and user prompt (changes in Appendix A.3).
    • Llama-Guard 2 and Granite Guardian 3.0: LLM safety classifiers; the former outputs safe/unsafe, the latter is fine-tuned from Granite-3.0-8B-Instruct and outputs yes/no.

    Evaluation and Resource Measurement

    • Metrics: AUC, ACC, F1, Recall, Precision, and per-dataset TP rate and FP rate heatmaps (Figure 1–4).
    • Resource overhead: memory usage, latency, throughput, and hardware are recorded (Table 5); on 100 random prompts repeated 10 times, batch size 1, hardware is a 10-core Xeon Gold 6258R CPU, 50GB RAM, V100 32GB GPU.
  4. What experiments did the paper conduct?

    15 guardrails are compared uniformly on the ID/OOD evaluation sets, with additional per-dataset TP/FP and memory, latency, and throughput measurements.

    Experimental Setup

    • Guardrails evaluated: 15 in total — Random Forest, BERT, DeBERTa, GPT2, Protect AI (v2), Llama-Guard 2, Langkit Injection Detection, SmoothLLM, Perplexity, OpenAI Moderation, Azure AI Content Safety, Nemo Inspired Input Rail, Langkit Proactive Defence, Vicuna-13b-v1.5 Refusal Rate, Granite Guardian 3.0.
    • Datasets: 27 (Table 1); the in-distribution evaluation set has 11387 entries (9543 benign, 1844 malicious), and OOD has 10327 benign plus 376 malicious.
    • Metrics: AUC, ACC, F1, Recall, Precision; per-dataset TP/FP rate heatmaps.
    • Decision method: the Vicuna baseline uses refusal keyword matching; OpenAI Moderation takes the highest probability among 11 harmful categories as binary classification; the paper does not report a unified external judge model.
    • Resource measurement: 100 random prompts repeated 10 times, batch size 1 (Table 5).
    • Note: the GCG attack was computed against Vicuna 13b (Figure 1 caption).

    Main Results

    In-distribution test set results (according to Table 2):

    Swipe horizontally to read the table

    GuardrailAUCACCF1RecallPrecision
    Random Forest0.990.9380.7620.6180.995
    BERT0.9940.9820.9430.9170.97
    DeBERTa0.9960.9910.970.9630.978
    GPT20.9940.9740.9160.8590.98
    Protect AI (v2)0.5690.8680.3480.2170.876
    Llama-Guard 2-0.950.8260.7330.946
    Granite Guardian 3.0-0.9710.9110.9160.905
    Perplexity-0.7710.1140.0910.153

    OOD dataset results (according to Table 3):

    Swipe horizontally to read the table

    GuardrailAUCACCF1RecallPrecision
    Random Forest0.9510.9790.5650.3960.987
    BERT0.9450.9620.5780.7470.471
    DeBERTa0.9760.9690.6720.910.533
    GPT20.9690.9790.720.7550.688
    Protect AI (v2)0.7720.970.5610.5510.572
    Llama-Guard 2-0.9610.4740.4970.453
    SmoothLLM-0.7990.1630.5590.096
    Perplexity-0.8240.0010.0030.001

    Each table omits 7 guardrails (including the worst-performing Nemo Inspired Input Rail, OOD ACC 0.38, F1 0.056).

    • The authors state that the BERT, DeBERTa, and GPT2 classifiers achieve high AUC and ACC on the test set and generalize competitively to new datasets, surpassing more computationally expensive open-source and closed-source defenses.
    • The authors state that Random Forest is competitive on ID test data but generalizes poorly OOD, although its computation and training cost are extremely low, and it can be continually updated with new data.
    • The authors state that LLM-based guardrails generally improve detection rates at the cost of higher FP rates, and that the increase in FP is uneven across datasets.

    Per-Dataset TP/FP Heatmaps

    • TP rate (Figure 1, 3): detection rates of guardrails vary greatly across jailbreak datasets, e.g., on AutoDAN, Random Forest and Protect AI are 0.00, while BERT and DeBERTa are 1.00; on the OOD MaliciousInstruct, Azure is 0.00 and NeMo is 1.00.
    • FP rate (Figure 2, 4): SmoothLLM has FP 0.35 on BoolQ and 0.17 on XSTest; NeMo Inspired Input Rail reaches FP 0.94 on XSTest and 0.34–0.68 on multiple OOD benign datasets.
    • Authors' interpretation: the authors state that the evaluation in the original SmoothLLM paper did not expose its low performance on benign datasets, showing that defenses must be evaluated with a broad set of datasets.

    Guardrails Without Training Control (Combined Dataset)

    • Results (Table 4): Granite Guardian 3.0 ACC 0.960, Recall 0.921; OpenAI Moderation ACC 0.086, Recall 0.845, Precision 0.086; Perplexity F1 0.070.
    • Authors' interpretation: for such guardrails the ID/OOD split does not apply, so results are reported on the combined data.

    Resource Overhead

    • Memory (Table 5): from 0.091 GB for LangKit Injection Detection to 26.03 GB for Vicuna-13b; BERT, DeBERTa, and GPT2 are 0.371–0.548 GB.
    • Extra inference calls: the authors state that SmoothLLM adds up to 10 inference calls per prompt and NeMo adds 1; SmoothLLM latency is 17.979 s/sample, throughput 0.1237 samples/s.
    • Authors' interpretation: the authors consider that increasing computation by an order of magnitude to defend against jailbreaks may not be feasible compared with lighter, underexplored solutions.

    Other Ablations and Analyses

    • NeMo-style guardrails relative to the Vicuna baseline: the authors state that detection and refusal performance improved, but the FP rate is also higher (Section 6).
    • The paper does not report statistical significance tests; it does not report the denominator details for each guardrail on OOD malicious samples (the 376 malicious OOD samples are the total).
  5. What are some points that can be further explored?

    The authors only point out the limitation that the evaluation data needed to be subsampled; the experiments cover 15 guardrails and 27 datasets.

    Limitations pointed out by the authors and future directions

    • Evaluation data subsampling: the authors point out in Section 7 (Conclusion and Limitations) that because some defenses are computationally expensive and the number of evaluation datasets is large, the final evaluation data had to be subsampled, which is the main limitation of this paper.
    • The paper does not list more future directions in the form of "future work"; in the RQ1 discussion in Section 6, the authors propose two possibilities: existing public data is already sufficient to represent jailbreak diversity, in which case a simple classifier is competitive; or the current academic literature lacks sufficiently rich data to demonstrate the generalization advantage of LLM-type defenses.

    Scope of experimental coverage

    • There are 15 guardrails tested, covering both detection-based and LLM-as-judge types (Table 2–4).
    • There are 27 datasets in total; the in-distribution evaluation set has 11387 entries (9543 benign, 1844 malicious), and OOD has 10327 benign plus 376 malicious (Section 5).
    • Resource measurements were repeated 10 times on 100 random prompts with batch size 1, on a V100 32GB GPU (Table 5).
    • The ID/OOD split applies only to the authors' self-trained models (Random Forest and Transformer classifiers); the remaining guardrails are reported on the combined data (Sections 5, 6).
    • The paper does not report statistical significance tests, nor human evaluation or rater model agreement; the GCG attack was computed only for Vicuna 13b (Figure 1 caption).
  6. Summarize the main content of the paper

    The authors benchmark 15 jailbreak guardrails and find that performance varies greatly with attack style, and that simple classifier baselines have competitive OOD performance.

    This is a benchmark evaluation paper on LLM jailbreak guardrails, from IBM Research, published at the NeurIPS 2024 Safe Generative AI Workshop.

    • Problem: The increase in jailbreak styles makes external guardrails a necessary defense line beyond alignment, but the evaluation protocols, datasets, and metrics of different defenses are scattered, there is a lack of systematic benchmarks specifically for guardrails, and performance under OOD attacks is unclear.
    • Method: 27 malicious and benign datasets are pooled and split into in-distribution (18) and OOD (9); 15 guardrails are evaluated uniformly with AUC, ACC, F1, Recall, and Precision, and memory, latency, and throughput are measured.
    • Main results: The authors report that guardrail performance varies greatly with jailbreak style; self-trained BERT, DeBERTa, and GPT2 classifiers perform comparably to more expensive open-source and closed-source guardrails on OOD data (DeBERTa OOD AUC 0.976, Recall 0.91, Table 3).
    • Cost trade-offs: LLM-based guardrails have higher detection rates but also higher FP rates; NeMo Inspired Input Rail reaches FP 0.94 on XSTest (Figure 2); SmoothLLM adds up to 10 inference calls per prompt, with latency 17.979 s/sample (Table 5).
    • Conclusion: The authors believe there is no one-size-fits-all guardrail, and the choice should depend on the defender's resources, the model's own defenses, and the threat spectrum faced; increasing computation by an order of magnitude may not be more feasible than lightweight solutions.
    • Limitations: The authors note that due to computational cost and dataset size, the final evaluation data were subsampled.

Related papers

Show 3 more papers
Read originalarxiv.org