Skip to content

Attack Atlas: A Practitioner's Perspective on Challenges and Pitfalls in Red Teaming GenAI

Attack Atlas: A Practitioner's Perspective on Challenges and Pitfalls in Red Teaming GenAI

arXiv· arXiv:2409.15398· Ambrish Rawat, Stefan Schoepf, Giulio Zizzo, Giandomenico Cornacchia, Muhammad Zaid Hameed, Kieran Fraser, Erik Miehling, Beat Buesser, Elizabeth M. Daly, Mark Purcell, Prasanna Sattigeri, Pin-Yu Chen, Kush R. VarshneyIBM Research; University of CambridgeIndexed by this site · 23 days ago· Originally publishedTools

Quick read

Other

Based on the paper PDF (AI-generated); refer to the original

The authors survey GenAI red teaming and blue teaming challenges from a practitioner perspective, propose a single-turn prompt attack taxonomy, Attack Atlas, and evaluate the TPR/FPR of 5 guardrails on 19 datasets.

Questions
GenAI red teaming and blue teaming lack practitioner-oriented guidance: academic surveys mostly characterize risks, but there is still no actionable framework for how to assess and mitigate prompt injection and jailbreaks in real environments, or how to develop and evaluate defenses.
Methods
The authors survey open problems in red teaming and blue teaming from an industry perspective, propose Attack Atlas—which classifies single-turn input attacks into 5 major categories and about 40 subcategories by the syntax, form and semantics of the prompt—and benchmark the TPR/FPR of 5 guardrails on 19 datasets.
Experiments and results
Guardrail evaluation shows performance varies greatly with the dataset's attack type: Vicuna-7b's TPR ranges from 0.24 on jailbreak prompts to 0.97 on harmful behaviours; the fine-tuned BERT remains competitive on out-of-distribution datasets but has a higher FPR on xstest boundary samples (0.29).
Limitations and future work
The authors note that Attack Atlas covers only single-turn attacks, with multi-turn and multimodal attacks left for future work; due to compute constraints, evaluation subsamples only 200 instances per dataset; the authors argue that existing benchmarks cover only a small fraction of possible input variations, and that open-ended rigorous benchmarks remain difficult.
Experimental setupBERT、SmoothLLM et al. · aart、attaq et al. · True Positive rate (TPr)、False Positive rate (FPr) et al.
Threat model
The paper discusses single-turn prompt attacks: jailbreak (bypassing all of an LLM's safety measures and alignment), direct injection (instructions in the user prompt override the system prompt) and indirect injection (instructions enter indirectly via external data such as websites into integrated applications such as RAG); red team practitioners often face black-box scenarios.
Evaluated models
BERTSmoothLLMVicuna-7bAzure AI C.S.Llama-Guard 2
Benchmarks
aartattaqdo not answergandalf_ignore_instructionsGCGharmful_behavioursjailbreak promptssaptapxstesttoxicchatmalicious_instructalpacaawesome_chatgpt_promptsboolqno robotspuffinsuper_natural_instructionsultrachat
Metrics
True Positive rate (TPr)False Positive rate (FPr)cosine similarity
AI summaryThe paper proposes the Attack Atlas framework, providing practitioners with a practical method for analyzing single-turn input attacks on generative AI. The authors note that as large language models enter production applications, natural language and multimodal systems present new attack surfaces, while guidance for practitioners to evaluate and mitigate these challenges in real-world settings remains limited. The paper surveys red team and blue team strategies for securing generative AI, identifies key challenges and open problems in defense development and evaluation, and aims to bridge the gap between academic research and practical security measures.

The paper proposes the Attack Atlas framework, providing practitioners with a practical method for analyzing single-turn input attacks on generative AI. The authors note that as large language models enter production applications, natural language and multimodal systems present new attack surfaces, while guidance for practitioners to evaluate and mitigate these challenges in real-world settings remains limited. The paper surveys red team and blue team strategies for securing generative AI, identifies key challenges and open problems in defense development and evaluation, and aims to bridge the gap between academic research and practical security measures.

Paper analysis

6 questions; open one to read the full answer

  1. What problem does this paper attempt to solve?

    GenAI red and blue teaming lacks practitioner-oriented guidance; the authors propose a challenge checklist and a single-turn attack taxonomy, Attack Atlas.

    Core problem: there is a gap between the academic characterization of GenAI adversarial risks and the actionable guidance practitioners need to assess and mitigate these risks in real environments.

    • Scenario and importance: As LLMs enter production applications, new attack surfaces arise in natural language and multimodal systems; the authors note that these threats are characterized by a low attack barrier (only a keyboard and human creativity are needed), and that LLMs cannot distinguish between the system-provided and user-provided parts of an input prompt.
    • Shortcomings of existing work: The authors state that existing academic surveys characterize GenAI adversarial risks but lack practitioner-oriented guidance for understanding and quantifying risk, addressing common threats, and selecting or developing appropriate defenses.
    • Core observation: The authors argue that red-teaming and blue-teaming practice have reached a fork—academic research focuses on sophisticated attacks and defenses, while practitioners care more about defending against low-cost, high-likelihood, high-severity attacks under budget-constrained environments.
    • Contributions of the paper: first, examining red-teaming and blue-teaming strategies from a practitioner perspective and comparing them with traditional adversarial machine learning; second, listing open problems and challenges in defense development, evaluation methods and benchmarks; third, proposing Attack Atlas, a taxonomy of single-turn input attacks organized by attack style (Figure 1).
    • Threat models (attack types discussed in the paper, Section 2):
      • Jailbreak: bypassing all of an LLM's safety measures, including alignment, to make it generate unsafe or unaligned outputs.
      • Direct Injection: the user prompt directly contains instructions that override the task defined by the system prompt, not necessarily requiring bypassing safety measures.
      • Indirect Injection: instructions enter indirectly through data such as websites, source code, or the outputs of other LLMs into integrated applications such as RAG.
      • Attacker knowledge: the authors note that in practice red teaming often faces black-box scenarios, which limits the available tools.
  2. What related research exists?

    Related work falls into four groups: GenAI risk surveys, automated red-teaming methods, guardrails and defenses, and existing benchmarks and datasets.

    GenAI risk surveys and taxonomies

    • Academic surveys [5, 43, 62, 65]—characterize the adversarial risks of GenAI; the authors note they lack practitioner-oriented guidance.
    • Taxonomies from misuse and risk perspectives [69, 57, 72]—characterize threats from misuse risks of frontier models (bias, toxic data generation, etc.); the authors note that malicious goals such as denial of service and SQL injection in complex workflows such as RAG are often missing from such characterizations.
    • Traditional adversarial machine learning—focuses on evasion [8], poisoning [9], inference [23], extraction [20], mainly with bounded perturbations in domains such as images and clear outcomes such as misclassification; the authors note that GenAI has a larger input-output space and that undesirable outcomes are more subjective and context-dependent.

    Automated red teaming methods

    • GCG [80], TAP [40], PAIR [11], AutoDAN [36] and the tool PyRIT [6]—automatically synthesize attack vectors; the authors note they differ in resource requirements and attack-type coverage, the generated attacks are not necessarily transferable, and the artifacts are affected by contextual factors such as the target model, seed data and priming of LLM components.
    • Samvelyan et al. [53] (Rainbow Teaming)—steers attacks within a matrix of attack styles and harm categories to ensure coverage; the authors note it is limited to the given attack styles and harm categories.
    • Apruzzese et al. [3]—note that real attackers favor simple, cheap, high-severity attacks rather than academic-style complex attacks.

    Guardrails and defenses

    • Various forms of input guardrails: score-based filtering (e.g., perplexity [27]), similarity detection, ML classifiers [26, 1], in-context learning [63], probing [50, 49, 14] and decoding techniques [24], differing in scale, latency, throughput and performance.
    • The authors note that current guardrails typically use a one-size-fits-all model to defend against all attack types [26] and cannot capture differences among threats; they also note that using ML methods to moderate LLM inputs and outputs may have fundamental limitations [19].

    Benchmarks and datasets

    • Safety dataset catalogs [52, 64] and benchmarks such as JailbreakBench [12], HarmBench [39], EasyJailbreak [78]; the authors note that existing benchmarks have narrow evaluation scope, and [12] contains only a few hundred samples.
    • The authors' own guardrail evaluation (Table 1, Table 2) uses 19 datasets (listed in Table 4), and the guardrails evaluated are BERT (fine-tuned), SmoothLLM, Vicuna-7b, Azure AI Content Safety, Llama-Guard 2.

    The authors position this paper as: complementing academic surveys with an industry perspective, unifying the characterization of single-turn input attacks with Attack Atlas, and serving as a starting point for practitioners to set up red and blue teaming operations.

  3. How does the paper address this problem?

    Surveys open problems in red teaming and blue teaming from a practitioner perspective, proposes a single-turn attack taxonomy organized by attack style, Attack Atlas, and conducts a guardrail benchmark evaluation.

    The paper consists of three parts: an analysis of open problems for red teaming and blue teaming, the Attack Atlas single-turn attack taxonomy, and a set of guardrail benchmark evaluations.

    Red teaming open problems (Section 2.1)

    • Scope and purpose: Attack goals depend on deployment context, organizational policy and regulation; the authors note that most red-teaming work uses only the limited harmful goals in AdvBench, often as few as 50 samples [80, 13], and that simple rewording can raise attack success rates [64].
    • Automation and scaling: The authors note that the attack vectors of automated tools (GCG, TAP, PAIR, AutoDAN, PyRIT) are affected by the target model and context, and that model updates can substantially reduce ASR; PAIR and TAP suffer from insufficient diversity and deviation from the goal.
    • Evaluation strategy: The authors note that academic red teaming targets single attack success or maximizing ASR, which is inconsistent with assessing real harm; keyword detection may have high false positive and false negative rates [34], and LLM-as-a-judge also has limitations [77, 34].

    Blue teaming open problems (Section 3.1)

    • Guardrail scope: The authors argue that defenses need to intercept attack intent, not only successful attacks; guardrail policy should be based on the typical attack behavior expected for the application.
    • Guardrail construction: The authors advocate customizing guardrails for different attack types and first specifying functional requirements (the boundary of allowable inputs, sufficient sampling) and non-functional requirements (context length, latency, throughput, memory); they note that some attacks can be handled by encoder-only models of about 100M parameters.
    • Guardrail evaluation: The authors emphasize the trade-off between protection and utility, the need to test guardrails with benign prompts, and out-of-distribution samples and exaggerated safety evaluation for ML classifiers.

    Attack Atlas taxonomy (Section 4)

    • Basis for classification: Classification is based only on the syntax, form and semantics of the prompt (surface features and underlying intent), not on prompt source (synthetic or human) or implied harm; it covers only single-turn attacks.
    • Five major categories (Figure 1, Table 5): Direct Instructions, Encoded Interactions, Social Hacking, Context Overload, Specialized Tokens, subdivided into about 40 subcategories (e.g., Payload-Splitting, N-shot, Role-playing Scenario, Token Smuggling); attackers can mix multiple categories (Mixed Technique).
    • Motivation: The authors report low cosine similarity between datasets (Figure 4) and high similarity within datasets (Figure 2b), and that grouping by attack style raises within-group similarity (Figure 2a); on this basis the authors argue that a unified characterization by attack style is useful.

    Guardrail benchmark evaluation procedure (Section 3.1.3)

    • Procedure: Evaluation on 19 datasets; BERT is fine-tuned on 60% of each dataset, with 20% validation and 20% test; due to compute constraints, 200 prompt instances per dataset are subsampled for evaluation.
    • Systems evaluated: three general purpose detectors (SmoothLLM, Vicuna-7b, Azure AI Content Safety, Llama-Guard 2, which the paper calls "three" but the table lists four) and fine-tuned BERT; toxicchat and malicious_instruct serve as out-of-distribution datasets that BERT was not trained on.
    • Metrics: True Positive rate on attack datasets (Table 1) and False Positive rate on benign datasets (Table 2).
  4. What experiments did the paper conduct?

    Evaluate TPR/FPR of 5 guardrails on 19 datasets, and analyze between-dataset and within-dataset differences using cosine similarity.

    Experimental setup

    • Guardrails evaluated: fine-tuned BERT, SmoothLLM, Vicuna-7b, Azure AI Content Safety (abbreviated Azure AI C.S. in the tables), Llama-Guard 2 (Table 1, Table 2).
    • Datasets: 19 in total (Table 4), of which 12 are attack/jailbreak datasets (aart, attaq, do not answer, gandalf_ignore_instructions, GCG, harmful_behaviours, jailbreak prompts, sap, tap, xstest, toxicchat, malicious_instruct) and 7 are benign datasets (alpaca, awesome_chatgpt_prompts, boolq, no robots, puffin, super_natural_instructions, ultrachat, with xstest additionally containing benign samples).
    • Evaluation procedure: BERT is fine-tuned with a 60% training / 20% validation / 20% test split; due to compute constraints, 200 prompt instances per dataset are subsampled for evaluation; toxicchat and malicious_instruct are out-of-distribution data for BERT.
    • Metrics: True Positive rate is reported on attack datasets (Table 1), False Positive rate on benign datasets (Table 2); the similarity analysis uses cosine similarity (Figures 2, 3, 4).
    • Judging method and number of repetitions: The paper does not report the specific judge model or threshold used to determine attack success, nor the number of repetitions.

    Main results

    According to Table 1 (TPR on attack datasets) and Table 2 (FPR on benign datasets), selected datasets:

    Swipe horizontally to read the table

    GuardrailGCG (TPR)jailbreak prompts (TPR)harmful behaviours (TPR)toxicchat (TPR, OOD)xstest (FPR)boolq (FPR)
    BERT0.990.820.920.710.290.00
    SmoothLLM0.810.290.980.470.170.39
    Vicuna-7b0.010.240.970.340.030.05
    Azure AI C.S.0.000.790.010.560.000.00
    Llama-Guard 20.840.030.980.150.000.01
    • The authors note that defensive performance varies greatly with the dataset's attack type, indicating that evaluation needs breadth: e.g., Vicuna-7b's TPR ranges from 0.24 on jailbreak prompts to 0.97 on harmful behaviours.
    • The authors note that specialized classifiers like BERT perform competitively even on out-of-distribution datasets and are lighter-weight than LLM-based guardrails; however, it has a higher FPR on xstest, which the authors attribute to xstest specifically checking edge cases, which large models handle better due to more thorough pretraining.
    • Azure AI C.S. has a TPR of 0.00 on aart, attaq, do not answer, GCG, sap, malicious_instruct, but 0.87 on gandalf_ignore_instructions and 0.79 on jailbreak prompts (Table 1).

    Dataset similarity analysis

    • What was measured: cosine similarity is used to measure similarity between and within prompt sets (Figures 2, 3, 4).
    • Results: Figure 4 shows low similarity between standard literature datasets (most values between 0.00–0.20, with a few such as jailbreakchat and chatgpt_dan at 0.42); Figure 2b shows higher similarity within datasets (the median for each dataset is mostly in the 0.10–0.30 range); Figure 2a shows that within-group similarity increases after merging by Attack Atlas category.
    • Authors' interpretation: datasets differ greatly from each other and are similar internally, indicating that a unified characterization of attacks by attack style (Attack Atlas) is useful; Appendix A.1 states that this supports viewing these datasets through the lens of Attack Atlas.

    Other ablations and analyses

    • The paper does not report other ablation experiments; Table 4 gives the mapping of the 19 datasets to Attack Atlas categories, and Table 5 gives the definition of each attack category and reference datasets.
    • Negative results: the authors note that automated red-teaming methods such as PAIR and TAP suffer from insufficient attack diversity and deviation from the goal (Section 2.1.2, cited from the literature rather than experiments in this paper).
  5. What are some points that can be further explored?

    The authors note that the taxonomy covers only single-turn attacks and needs to be extended to multi-turn and multimodal; due to compute constraints, evaluation uses only 200 instances per dataset.

    Limitations noted by the authors and future directions

    • Single-turn attacks only: Attack Atlas characterizes only single-turn attack strategies; the authors state that multi-turn [31, 42] and multimodal prompt attacks should be included to extend the dimensions of attack tactics (Section 4.1).
    • Limited benchmark coverage: The authors note that although the number of attack datasets is growing, they still cover only a small fraction of possible input variations and perturbations; the NLP field lacks an effective optimization process like PGD for searching jailbreak variants, and inputs are not subject to Lp-ball-style constraints, so open-ended rigorous benchmarks are difficult; focus should be on specific attacks with high likelihood and high severity (Section 3.1.3).
    • Evaluation methods not systematized: The authors note that scalable, customizable attack success metrics (e.g., refusal detection, content moderation) still lack systematization and benchmarks, and need to be adapted to the attack goal (Section 2.1.3).
    • Threat models need grounding: The authors suggest that GenAI threat models should be based on attacks that actually occur, and that tools and benchmarks should shift toward being informed by real attacks and resource constraints (Section 5).
    • The taxonomy will evolve: The authors note that as new attacks and defenses emerge, the threat taxonomy will evolve, and guardrail policy should be based on the typical attack behavior expected for the application (Section 3.1.1).

    Scope of experimental coverage

    • The guardrails evaluated are BERT (fine-tuned), SmoothLLM, Vicuna-7b, Azure AI Content Safety, Llama-Guard 2, 5 in total (Table 1, Table 2).
    • Evaluation uses 19 datasets: 12 attack/jailbreak datasets and 7 benign datasets, with xstest additionally containing both attack and benign samples (Table 4).
    • Due to compute constraints, 200 prompt instances per dataset are subsampled for evaluation; BERT is split 60%/20%/20% into training, validation and test (Section 3.1.3).
    • The paper does not report the judge model or threshold used to determine attack success, the number of repetitions, or measured latency or throughput for each guardrail.
    • The similarity analysis covers the 23 datasets listed in Figure 4 (including chatgpt_dan, deepset_prompt_injection, dolly, human_preferences, jailbreakchat, orca, piqa, production_jailbreak, prompt_extraction, prompt_injection_purple_llama, etc., which are not used in the evaluation).
  6. Summarize the main content of the paper

    A practitioner-perspective analysis of GenAI red teaming and blue teaming challenges, with the single-turn attack taxonomy Attack Atlas and an evaluation of 5 guardrails on 19 datasets.

    A practitioner-perspective paper from IBM Research et al. discussing the challenges of GenAI red teaming and blue teaming in real deployments, and proposing a single-turn prompt attack taxonomy, Attack Atlas.

    • Problem: The authors state that academic surveys characterize GenAI adversarial risks but lack practitioner-oriented guidance; academic red teaming pursues sophisticated attacks and high ASR, whereas practitioners need to defend against low-cost, high-likelihood, high-severity attacks under budget constraints.
    • Red teaming challenges: Attack goals depend on deployment context; the attack vectors of automated tools (GCG, TAP, PAIR, etc.) are sensitive to the target model and context and lack diversity; definitions and metrics of attack success (keyword detection, LLM-as-a-judge) lack consistent standards.
    • Blue teaming challenges: The authors argue that one-size-fits-all guardrails are unrealistic and should be customized by attack type, with functional and non-functional requirements specified first, and the trade-off between protection and utility tested with benign prompts and out-of-distribution samples.
    • Attack Atlas: classifies single-turn attacks by the syntax, form and semantics of the prompt into five major categories—Direct Instructions, Encoded Interactions, Social Hacking, Context Overload, Specialized Tokens—and about 40 subcategories (Figure 1, Table 5); the authors report low cosine similarity between datasets and high similarity within datasets, with within-group similarity increasing after grouping by attack style (Figures 2, 4).
    • Guardrail evaluation: 5 guardrails are evaluated on 19 datasets (200 instances subsampled per dataset); defensive performance varies greatly with attack type, e.g., Vicuna-7b's TPR ranges from 0.24 on jailbreak prompts to 0.97 on harmful behaviours (Table 1); fine-tuned BERT remains competitive on out-of-distribution datasets and is lighter-weight, but reaches an FPR of 0.29 on benign xstest edge cases (Table 2).
    • Authors' conclusion: GenAI threat models should be based on attacks that actually occur, and misuse/safety classifications should be complemented by a security-perspective attack taxonomy; Attack Atlas aims to provide the community with a unified starting point.

Related papers

Show 3 more papers
Read originalarxiv.org