AttackEval: A systematic empirical study of the effectiveness of prompt injection attacks against large language models

Existing work leans toward defenses; which injection strategies are effective on the attack side and why still lacks systematic comparison.

• Scenario: The authors state LLMs have entered customer service assistants, autonomous agents, code assistants, and retrieval-augmented pipelines. Prompt injection inserts adversarial text into inputs to override established behavior, and the authors state this vulnerability has no direct counterpart in classical computer security.
• Existing gaps: The authors state detection-based defenses achieve high accuracy on curated benchmarks but degrade on new variants, attributing this to an insufficiently characterized attack landscape. Existing attack research is mostly limited to single strategies or informally compiles user-shared jailbreak examples.
• Core hypothesis: The authors believe that without comparing which strategies work and why, defense designers cannot anticipate the correct threat model. The taxonomy is organized by primary evasion mechanism: surface form, sequence processing, alignment training.
• This paper proposes: AttackEval, containing ten categories and three groups, plus AttackEval-250 (25 handcrafted prompts per category, 250 total). Experiments report ASR across four defense tiers on a simulated task-constrained assistant. The authors state this is the first comprehensive, controlled empirical evaluation of prompt injection attack effectiveness.
• Evaluation target: The victim system is an email assistant whose system prompt restricts it to answering email-related queries. Success is defined by whether the defense lets the prompt through, i.e., the prompt is not filtered and reaches downstream. The paper does not write attacker knowledge in terms of white-box, black-box, gray-box, or no-box.

Prompt injection

• Perez and Ribeiro (2022): The authors state that this work first formally described prompt injection, using text instructions to override system-level instructions.
• Greshake et al. (2023): Extended injection to indirect injection in external content for RAG. The authors state that this work and Zhan et al. (2024) show that modern agent frameworks remain vulnerable.
• Wang et al. (2025): Divided prompt injection into System Prompt Forgery, User Prompt Camouflage, and Model Behavior Manipulation, and performed semantic defense. The authors state that this paper refines and extends the attack side of that taxonomy.

Jailbreak

• Gradient and automatic prompts: GCG (Zou et al., 2023), AutoDAN (Xu et al., 2024; Zhu et al., 2023), HotFlip (Ebrahimi et al., 2018), AutoPrompt (Shin et al., 2020). In the introduction, the authors treat "using only one strategy" (such as gradient-based adversarial suffixes) as too narrow in scope.
• Black-box methods: The paper classifies PAIR (Chao et al., 2023), TAP (Mehrotra et al., 2023), and many-shot jailbreaking (Anil et al., 2024) as black-box. Wei et al. (2023) analyzed why safety training fails using competing objectives.
• User-written jailbreaks: Shen et al. (2023) and Liu et al. (2023) characterized DAN-style prompts in the wild. The authors state that this paper focuses on prompt injection in deployed task-specific systems, not unconstrained jailbreaks.

Defenses

• Detection: Perplexity filtering (Jain et al., 2023), latent-space anomaly detection, and template-based methods (Wang et al., 2025).
• Prevention: SecAlign (Chen et al., 2024) uses preference optimization; DataSentinel (Zhang et al., 2024) uses a game-theoretic detection model.
• PromptSleuth (Wang et al., 2025): The authors state that it performs detection through task-level semantic intent reasoning. The authors state that this paper is complementary work, characterizing the attack surface these defenses need to cover.

Red teaming and this paper's position

• Red teaming: Perez et al. (2022), Ganguli et al. (2022), and Carlini et al. (2023) systematically probe LLM safety. The authors state that the difference is that this paper only looks at prompt injection on task-constrained system assistants, not general harmlessness violations.
• Baselines and benchmarks: The experiments did not run the above defenses or jailbreak algorithms as comparisons. The conditions are self-built four levels (no defense, L1, L2, L3); no defense is called by the authors the baseline representing the theoretical upper-bound ASR. The data is the self-built AttackEval-250.

The authors position this paper as refining the attack side of Wang et al. (2025), and restrict the comparison to task-constrained assistants rather than unconstrained jailbreaks.

The overall approach is AttackEval: first encode ten attack categories by their evasion mechanism, then use handcrafted prompts to measure a rule-based simulated assistant's filter pass rate; pairwise combinations are not tested item by item but estimated with a formula.

Classification and Data

Table 1 groups the ten categories into three groups.

• Syntactic (S): DO uses explicit meta-instructions to override system instructions; RI specifies a persona not bound by the original constraints; OBF uses encoding or glyph transformations to change surface form, with the paper naming Base64, Unicode homoglyphs, leetspeak, ROT13, and byte-level hexadecimal.
• Contextual (C): Table 1 lists IW, CT, PS. IW embeds the payload in JSON, XML, YAML, SQL comments, or Markdown; CT first injects a fake task-completion signal, then appends an unauthorized instruction; PS splits one injection across multiple turns or fragments. Section 3.1, however, discusses IW under the Syntactic heading.
• Semantic/Social (SS): EM wraps the payload in dilemmas or urgency; RF uses flattery, capability appeals, or promises of positive feedback; TC pressures with non-compliant negative consequences; NT rewrites it as fiction, hypothesis, or simulation. The authors state that EM exploits RLHF's helpfulness bias, RF exploits reward associations formed during training, TC triggers learned patterns of obedience to authority, and NT exploits the looser norms for fictional content in training.
• Scale: 25 items per category, 250 total, handcrafted, covering varied phrasings, lengths, syntactic styles, and degrees of explicitness. The authors state the writing follows published red-team research and jailbreak forums. The paper does not report the number of writers or consistency checks.

Victim System and Four Defense Tiers

The victim system is written as a production assistant with task constraints, with the role of an email assistant, and the system prompt restricts it to answering email-related queries only. The authors state the structure is the same as the scenarios in DataSentinel and PromptSleuth; Section 6.5 states the implementation is a rule-based simulation, not a live LLM.

1. No Defense: All inputs are accepted; the authors call this the baseline for the theoretical upper-bound ASR.
2. L1 Keyword: Keyword/regex blacklist matching common injection patterns. The authors state it represents the simplest deployed filter.
3. L2 Semantic: Adds structural anomaly detection on top of L1, including regexes for role-play cues, obfuscation markers (Unicode outliers, Base64-like sequences), and topic-deviation signals.
4. L3 Intent-Aware: Adds semantic intent analysis on top of L2. The authors state it imitates PromptSleuth's task-relationship reasoning, flagging non-email task intent, manipulative language, narrative framing, and instruction wrapping.

The paper does not give the regexes, thresholds, or model names for the intent classifier at each tier. The abstract states OBF can defeat semantic similarity checks; Section 4.1's text for L2 is structural anomaly detection, without mentioning similarity scores.

Success Criterion and Two Definitions

Success is the defense passing the input, i.e., the prompt reaches downstream and is not filtered, not whether the model produces harmful content. Category ASR is the pass rate:

ASR=1/N∑_i=1^N1[defense(p_i)=pass]

That is, the proportion of N prompts judged as pass. The 95% interval comes from bootstrap with B=200 for each category–defense pair. Stealth is 0.5·min(len(p)/20, 1) plus 0.5·(1−keyword density), with higher meaning more stealthy. ∆ASR is the combined ASR minus the better of the two strategies.

Estimation of Combined Attacks

Combination wraps a base prompt from category A into a template from category B. The paper states the combined ASR is estimated using the independence complement plus a synergy term, rather than measured item by item on the simulator:

ASR_{A⊕ B}=P_A+P_B−P_A· P_B+ϵ_AB

εAB takes values from 0.04 to 0.14; the authors state orthogonal mechanisms take high values and related mechanisms take low values. A total of C(10,2)=45 pairs are evaluated. The paper does not give εAB for each pair, nor does Section 4.4 state which tier PA and PB are taken from; Section 5.3's results correspond to the intent-aware defense.

Experimental Setup

• Victim side: A rule-based simulated email assistant, not a live LLM (Section 6.5). The paper does not give the model name, version, or API called. Figure 6 only uses the four tier labels Weak (7B base), Moderate (13B instruct), Strong (70B RLHF), SOTA (GPT-4 class).
• Data: AttackEval-250, 25 items per category across 10 categories, 250 total. Category codes are in Table 1.
• Conditions: No Defense, L1 Keyword, L2 Semantic, L3 Intent-Aware. No defense is the baseline the authors call the theoretical upper bound. L1–L3 are self-built approximations, not replications of PromptSleuth or DataSentinel.
• Metrics: ASR is the proportion of defense(pi)=pass. Also Stealth and ∆ASR. There is no LLM judge, nor human judgment of whether the model followed the instruction.
• Uncertainty: Bootstrap with B=200 for each category–defense pair, reporting 95% CI. The paper does not report the number of independent repetitions, nor does the main text give CI endpoints.
• Wording discrepancy: The abstract states OBF defeats both keyword matching and semantic similarity checks; Section 4.1's description of L2 is structural anomaly detection, without mentioning a similarity threshold.

Main Results

The table below is based on Table 2. Ten categories exceed 8 rows; CT, TC, NT centered at L3 are omitted; RI, the lowest at L3, is retained.

| Category | None | L1 | L2 | L3 | Group | | DO | 1.00 | 0.36 | 0.24 | 0.24 | S | | RI | 1.00 | 0.68 | 0.28 | 0.12 | S | | OBF | 1.00 | 0.84 | 0.72 | 0.76 | S | | IW | 1.00 | 0.24 | 0.24 | 0.16 | C | | PS | 1.00 | 0.64 | 0.52 | 0.52 | C | | EM | 1.00 | 0.56 | 0.56 | 0.44 | SS | | RF | 1.00 | 0.64 | 0.52 | 0.48 | SS |

According to Table 2, the omitted CT, TC, NT are 0.32, 0.32, 0.28 at L3 respectively; these three categories are also 1.00 under No Defense. The corresponding cells in the Figure 3 heatmap are the same as Table 2.

• The authors state OBF is the single-category attack most resistant to defenses, because it avoids keywords and partially avoids L2's regex anomaly detection; they explain the rise from L2 to L3 as intent analysis being unable to parse obfuscated text.
• The authors state EM and RF's natural-language surface looks like normal input. EM's row has the same L1 and L2, and the authors use this to claim semantic filtering has no marginal effect on emotional manipulation.
• The authors state RI's relative drop is the largest among the ten categories, giving an 82% relative reduction (from L1 to L3), and believe its persona phrasing is caught by semantic-deviation checks.

Combined Attacks

• What was tested: 45 pairwise combinations. Section 4.4 states ASR is estimated by the independence complement formula plus εAB, not measured item by item.
• Numbers in the main text: For the intent-aware defense, the authors list the top five as OBF+EM 0.976 (∆ASR=+0.216), OBF+RF 0.958 (+0.198), OBF+CT 0.941 (+0.181), OBF+PS 0.939 (+0.179), OBF+TC 0.912 (+0.152). The abstract writes OBF+EM as 97.6%.
• Figure-text inconsistency and interpretation: The right side of Figure 4 is labeled with +0.74, +0.70, +0.62, etc., inconsistent with the main text's ∆ASR; after text conversion they cannot be reliably mapped to categories, so they are not cited by category. The authors state all top five contain OBF, and describe the complementarity as: OBF handles structural/lexical checks, while the second component maintains semantic naturalness.

Stealth Correlations and Model Tiers

• Three inconsistent correlation coefficients: The abstract writes r=0.71 for residual ASR against semantic defenses. Section 5.4 writes r≈0.21 for L1 and r≈0.71 for L3, stating that as defenses strengthen, selection pressure pushes toward natural-language strategies. Figure 5 is labeled L1 r=0.551, L3 r=0.057; the caption still states L3 has a stronger positive correlation.
• Figure 6 average ASR: Weak (7B base) is labeled 1.00, Moderate (13B instruct) 0.56, Strong (70B RLHF) 0.43, SOTA (GPT-4 class) 0.35. The main text writes the weakly aligned average is close to 100%, SOTA drops to about 30%, and states OBF is 0.76, about 2.5 times the average.
• Authors' claim about the SOTA tier: Section 5.5 states DO and RI's ASR are 0.24 and 0.12, and states these two categories are effectively solved. The in-figure text of Figure 6 only labels each tier's average, without per-category readings. The paper does not state whether this section called real models of these sizes, nor whether it is the same experiment as Table 2's L3.

Other Ablations and Analyses

• The paper has no standalone ablation table. OBF goes from 0.72 at L2 to 0.76 at L3 (Table 2).
• EM's L1 and L2 are both 0.56; TC's L1 and L2 are both 0.48 (Table 2).
• IW's L1 and L2 are both 0.24, L3 is 0.16; DO's L2 and L3 are both 0.24 (Table 2).
• Section 5.2 states that under L3 only OBF, PS, EM, RF are above 40%; in Table 2 these four are 0.76, 0.52, 0.44, 0.48, and the rest do not exceed 0.32.
• εAB takes [0.04, 0.14]; the paper does not give each ε for the 45 pairs (Section 4.4).
• The Figure 1 caption states EM, RF, NT have larger footprints under strong defenses, while DO, RI shrink as defenses strengthen; the readings in the figure are not preserved in the text. The caption writes high ASR under L3 as the heatmap top row, but in the extracted text the topmost row is No Defense, with L3 at the bottom row.

Limitations and future directions noted by the authors

• Simulation rather than online model: The victim system is a rule-based simulation, not an online production LLM (Section 6.5).
• Behavioral detail: The authors state this may not capture all details of real model behavior (Section 6.5).
• Wording sensitivity: Real models may behave inconsistently across different phrasings even with the same intent (Section 6.5 a).
• Defenses are not the original systems: The four tiers approximate but do not fully reproduce PromptSleuth or DataSentinel (Section 6.5 b).
• Taxonomy may miss new vectors: The authors state that attack tooling is still evolving, the taxonomy may not cover all new vectors, and cite adversarial suffixes and injection via system context as examples (Section 6.5 c).
• Subsequent validation: The authors state these findings should subsequently be validated on online API deployments (Section 6.5).

Experimental coverage

• Evaluation data is AttackEval-250, 25 items per each of 10 categories; conditions are four self-built defense tiers; ASR is defined as defense(pi)=pass (Section 4, Table 2).
• The combination part computes all 45 pairs, with ASR estimated by the formula in Section 4.4; the paper does not report running combined prompts item by item on the simulator.
• Figure 6 labels the four-tier average ASR as 1.00, 0.56, 0.43, 0.35; the paper does not report the specific model names or APIs for these tiers, nor whether this is the same experiment as Table 2.
• Uncertainty is reported as 95% bootstrap with B=200 per pair; the paper does not report CI endpoints, significance tests, number of independent repetitions, human agreement, or defense false rejection rates.
• The victim task is written as a single email assistant setting (Section 4.1). Defenses are none, L1, L2, L3; the authors state in Section 6.5 that they approximate rather than reproduce PromptSleuth and DataSentinel.

AttackEval compares the pass rates of ten prompt injection categories through four defense tiers on a rule-simulated email assistant.

• Problem: The authors believe there are many defense papers but relatively few comparisons of attack strategies, so it is unclear which injection category subsequent defenses should cover.
• Approach: Ten categories are split into Syntactic, Contextual, Semantic/Social, with 25 handcrafted prompts per category, 250 in total. Defenses range from full allowance to keyword, structural anomaly, and intent analysis. Success only means the filter let the prompt through. ASR for the 45 pairs is estimated using independence complements plus synergy terms, not measured item by item.
• Single-category results: In Table 2, with no defense, all ten categories have ASR 1.00. Under L3, OBF is 0.76, PS is 0.52, RF is 0.48, EM is 0.44, RI is 0.12. OBF goes from 0.72 at L2 to 0.76 at L3; EM is 0.56 at both L1 and L2.
• Combinations and model tiers: The authors state the estimated OBF+EM under L3 is 0.976, and the abstract writes 97.6%. Figure 6 labels the weakly aligned tier average as 1.00 and the SOTA (GPT-4 class) tier average as 0.35; the main text also writes the SOTA average is about 30%, and states OBF is 0.76 in that tier. The stealth correlation coefficient is inconsistent across the abstract, Section 5.4, and the Figure 5 label.
• Authors' conclusions: The authors believe OBF arises from the asymmetry that the model can decode while defenses only look at the surface; EM and RF exploit helpfulness in alignment and do not necessarily trigger task-deviation detection. The authors suggest layered inspection, decoding or normalization before inspection, and handling emotional manipulation and sycophancy separately. Section 6.5 states these numbers come from a rule-based simulation and plans to re-validate with online APIs. The abstract also plans to release code and data after publication.