Skip to content

Translation in progress Some entries still include text in the original language.

Original
arXiv· arXiv:2601.07185· Shawn Li, Chenxiao Yu, Zhiyu Ni, Hao Li, Charith Peris, Chaowei Xiao, Yue Zhao·Indexed by this site · Originally publishedSelectedAttention42

Prompt attack defenses learn surface heuristic rules

Defenses Against Prompt Attacks Learn Surface Heuristics

Quick read

Analysis and theory

Based on the paper PDF (AI-generated); refer to the original

The authors analyze fine-tuning defenses such as StrucQ and SecAlign, claiming they learn surface shortcuts: the refusal rate for the suffix task is as high as 90%, and trigger words increase false refusals by up to 50%.

Questions
Supervised fine-tuning prompt injection defenses have high attack refusal rates on standard benchmarks, but the authors argue that they may only learn surface correlations in the defense data rather than recognizing malicious intent, thereby systematically falsely refusing benign inputs.
Methods
The authors propose three hypotheses—positional bias, token trigger bias, and topic generalization bias—and construct diagnostic datasets that change only a single surface factor while preserving benign semantics, measuring changes in refusal rate and accuracy on Llama 3, Mistral, and five prompt guards.
Experiments and results
The authors report: when benign tasks are concatenated, the suffix task refusal rate is up to 90% (Table 2); a single trigger word raises false refusals by up to about 50 percentage points (Table 3); after defense fine-tuning, average accuracy on the three reasoning benchmarks drops by up to 18.8 (Table 4); after removing trigger words, prompt guard refusal rates decrease (Table 5).
Limitations and future work
The authors state that the study is a focused empirical analysis under specific controlled conditions, and that the models, benchmarks, and defense methods are only a snapshot of current practice and do not cover all defense designs; the experiments cover only two open-source model families, two types of fine-tuning defenses, and five prompt guards.
Experimental setupLlama 3、Mistral et al. · GPQA (gpqa_diamond)、MMLU et al. · Refusal Rate (RR)、Accuracy (Acc) et al.
Threat model
The paper analyzes defensive behavior rather than proposing an attack: benign prompts enter the pipeline via user queries or externally retrieved content, which may contain adversarial instructions (prompt injection, including direct and indirect types); the defender performs supervised fine-tuning with data labeled benign/attack, and the authors hypothesize that the model learns surface features ϕ(x) rather than malicious intent.
Evaluated models
Llama 3MistralProtectAIv2LakeraGuardPromptGuard2DeepsetFMOPS
Benchmarks
GPQA (gpqa_diamond)MMLUAIME 2024PINTWildGuardInjectGuard / NOTINJECT
Metrics
Refusal Rate (RR)Accuracy (Acc)utility drop ∆utiltrigger gap ∆
AI summaryand selection rationaleSupervised fine-tuning-based prompt injection defenses rely on surface features rather than malicious intent, producing systematic false refusals on benign inputs. The researchers constructed a controlled-variable diagnostic set, evaluated it on two base models, Llama 3 and Mistral, and on fine-tuned defenses such as StruQ and SecAlign and various prompt guardrails, and summarized three types of shortcut behavior: position bias, trigger-word bias, and topic generalization bias. In concatenated prompts, the refusal rate for benign tasks placed later rose from below 10% to as high as 90%; inserting a single attack-related trigger word raised the false refusal rate by up to 50%, while matched non-trigger-word controls showed no such effect; on reasoning benchmarks such as GPQA, MMLU, and AIME, defense models' accuracy dropped by up to 40%. Trigger-word ablation showed that removing the trigger word substantially lowered guardrail refusal rates, providing causal evidence that refusals are driven by surface word form; the authors argue that aggregate refusal metrics are insufficient to measure defense reliability, and the code and data are open-sourced on GitHub.

Supervised fine-tuning-based prompt injection defenses rely on surface features rather than malicious intent, producing systematic false refusals on benign inputs. The researchers constructed a controlled-variable diagnostic set, evaluated it on two base models, Llama 3 and Mistral, and on fine-tuned defenses such as StruQ and SecAlign and various prompt guardrails, and summarized three types of shortcut behavior: position bias, trigger-word bias, and topic generalization bias. In concatenated prompts, the refusal rate for benign tasks placed later rose from below 10% to as high as 90%; inserting a single attack-related trigger word raised the false refusal rate by up to 50%, while matched non-trigger-word controls showed no such effect; on reasoning benchmarks such as GPQA, MMLU, and AIME, defense models' accuracy dropped by up to 40%. Trigger-word ablation showed that removing the trigger word substantially lowered guardrail refusal rates, providing causal evidence that refusals are driven by surface word form; the authors argue that aggregate refusal metrics are insufficient to measure defense reliability, and the code and data are open-sourced on GitHub.

Why it was selectedThe paper uses controlled-variable diagnostics to show that prompt injection defenses rely on surface features such as position and trigger words; teams deploying defenses can use this to assess false refusals and utility loss.

Paper analysis

6 questions; open one to read the full answer

  1. What problem does this paper attempt to solve?

    The authors say fine-tuning-based prompt injection defenses learn surface shortcuts rather than malicious intent, systematically falsely refusing benign inputs.

    Core question: Do supervised fine-tuning (SFT)-based prompt injection defenses truly recognize malicious intent, or do they rely only on surface correlations in the defense data to refuse?

    • Problem setting: LLMs are deployed in safety-sensitive scenarios such as customer service, RAG, and agent automation, where they need to follow system instructions while completing benign requests; when adversarial instructions enter the pipeline through user queries (direct) or external retrieved content (indirect), the model may execute the injected commands, and the authors believe a single safety failure can trigger real-world actions.
    • Shortcomings of existing methods: The authors say the training data of fine-tuning defenses such as StrucQ and SecAlign is dominated by explicitly injected samples (patterns such as instruction override and delimiter escape), and the models' high attack refusal rates on standard benchmarks are often taken as evidence of robustness, but empirical behavior shows they systematically falsely refuse safe inputs.
    • Core hypothesis: When the correlation between surface features ϕ(x) and labels is much higher than that between semantic intent and labels, empirical risk minimization will lead the model to learn the rule "ϕ(x) present means refuse," even if the prompt is entirely benign (Section 3.1).
    • The paper proposes: Three testable shortcut bias hypotheses (positional bias, token trigger bias, topic generalization bias), accompanying controlled diagnostic datasets, and a systematic evaluation on two base models and multiple defense pipelines.
    • Analysis target (not an attack): The paper does not propose new attacks but measures defender behavior—the victim systems are defense fine-tuned open-source LLMs (Llama 3, Mistral instruction versions) and external prompt guard classifiers; all inputs are benign prompts, changing only a single factor of position, token, or topic.

  2. What related research exists?

    Related work falls into three groups: prompt injection attacks, model-level defenses, and over-defense observations; this paper positions itself as a diagnostic analysis of defense shortcut behavior.

    Prompt injection attacks

    • Perez and Ribeiro (2022)—proposed direct prompt injection, where attackers directly inject malicious instructions through user-visible input; Willison (2023) discussed the limitations of delimiter-type techniques.
    • Greshake et al. (2023), Chen et al. (2024, 2025b)—indirect prompt injection, embedding adversarial instructions in external retrieved content such as web pages and documents.
    • Liu et al. (2024), Yi et al. (2025)—formalized and benchmarked prompt injection attacks and defenses (Appendix A also mentions variants such as template escape and completion-style attacks).

    Model-level defenses

    • StruQ (Chen et al., 2025a)—fine-tunes the model with a structured prompt format that preserves delimiters, distinguishing instruction fields from data fields; SecAlign (Chen et al., 2024)—constructs preference pairs for preference optimization; Meta SecAlign (Chen et al., 2025b)—extends this line to foundation models. This paper uses StrucQ and SecAlign as representatives of fine-tuning defenses for diagnosis.
    • Constitutional AI (Bai et al., 2022) and deliberative alignment (Guan et al., 2024)—have the model explicitly reason about safety constraints during generation.
    • Wallace et al. (2024), Wu et al. (2024)—instruction hierarchy, training the model so that system-level instructions take precedence over user input.

    Prompt guards and over-defense

    • ProtectAIv2, LakeraGuard, PromptGuard2, Deepset, FMOPS—external filters that classify before input reaches the model; this paper treats them as evaluation targets.
    • Li and Liu (2024) (InjecGuard/NOTINJECT)—benchmarked and mitigated over-defense in guard models; Han et al. (2024a) (WildGuard)—an open safety moderation tool. The paper says its shortcut failures are consistent with these over-defense observations and directly uses their benign data.

    Positioning of this paper: The authors say prior work has noted the safety–utility trade-off; this paper's contribution is analyzing how supervised defense fine-tuning produces shortcut behavior, manifested as false refusals, reasoning suppression, and generalization degradation on benign inputs (Appendix A.2).

  3. How does the paper address this problem?

    Proposes three shortcut bias hypotheses, constructs benign diagnostic sets that change only a single surface factor, and measures defense behavior using refusal rate and accuracy drop.

    整体思路：把"防御是否依赖表面线索"形式化为三个可证伪假设，为每个假设构造语义保持良性、只改变一个表面因素的受控诊断集，再比较基座模型与防御微调模型的拒答率（RR）和准确率（Acc）。

    问题形式化

    • 防御模型 fθ′ 由基座模型在标注数据集 Ddef = {(xi, zi)}（zi ∈ {benign, attack}）上监督微调得到；理想行为是对注入输入高拒答、对良性输入（含分布外良性）低拒答且准确率不掉。
    • 核心不等式：当 I(ϕ(x); z) ≫ I(semantic intent(x); z) 时，训练目标可仅靠表面特征 ϕ(x) 最小化，模型学到"ϕ(x) 出现即拒答"。

    三个失败假设

    • H1 位置偏置：拼接提示词 x = [xA; xB] 两段均良性时，P(reject | xB) 仍很高，即模型把提示词后段当作可疑区域。
    • H2 token 触发偏置：对良性 x，插入攻击数据中高频出现的字符串 t ∈ T 后，P(reject | t∈x) > P(reject | t∉x)。
    • H3 话题泛化偏置：对来自未见话题域 Ttest 的良性提示词，防御模型准确率低于训练话题域 Ttrain，即 Ex∼ Ttest[Acc(fθ′(x))] < Ex∼ Ttrain[Acc(fθ′(x))]。

    诊断数据集设计

    • Position Stress Set（H1）：从 GPQA、MMLU、AIME 2024 随机配对同基准的两道良性题拼接；另有五题拼接消融（MMLU，173 组）。
    • Keyword Trigger Set（H2）：基线安全集为 PINT 良性部分与 WildGuard 的 971 条良性请求混合；触发压力集用 InjectGuard/NOTINJECT 的三词触发子集（每个触发计数档 113 条良性提示词）；另生成插入相似长度随机技术词的对照 x+r，预期 P(reject|x+t) ≫ P(reject|x) ≈ P(reject|x+r)。
    • Topic Generalization Set（H3）：不拼接、不插触发词，直接在 GPQA/MMLU/AIME 单题上测准确率；理由是这些学术推理域在防御微调语料（以指令覆盖、分隔符操纵等元指令话题为主）的支持集之外。

    评测指标

    • RR：H1 下按答案可提取性判定（无法提取有效答案即记拒答）；H2 下模型做注入二分类，判为 unsafe/injection 即记拒答；prompt guard 判 unsafe 同样记拒答。
    • 偏置度量：H1 比较 RR(xA) 与 RR(xB)；H2 计算 ∆ = RRtrigger − RRbaseline；H3 报告相对基座的准确率落差 ∆util = Accdef − Accbase。
    • 生成模型评测用 greedy decoding（temperature 0），H2 采用 Li and Liu (2024) 的注入检测提示协议（附录 B.3.1 给出完整提示词）。

  4. What experiments did the paper conduct?

    Three biases are validated on Llama 3, Mistral, and five prompt guards: the refusal rate for the suffix task is as high as 90%, and trigger words increase false refusals by up to about 50 percentage points.

    实验设置

    • 被测模型：两个开源指令模型家族 Llama 3 与 Mistral（附录 B.3 写明为 Llama-3-8B、Mistral-7B），各自比较 Base、+StrucQ、+SecAlign 三种状态；另有五个 prompt guard：ProtectAIv2、LakeraGuard、PromptGuard2、Deepset、FMOPS。
    • 数据集与规模：GPQA gpqa_diamond 全部 100 题、MMLU 验证集 1,204 题（57 个域）、AIME 2024 全部 30 题；H2 基线安全集为 PINT 良性部分（正文写 3,007 条，附录写约 2.8k）加 WildGuard 971 条，触发压力集为 InjectGuard/NOTINJECT 三词触发子集（每档 113 条）。
    • 指标：RR（拒答率，越低越好）、Acc（准确率，越高越好）、括号内为相对基座的落差；H2 报告 ∆ = RRtrigger − RRbaseline。
    • 判定方式：H1 按答案可提取性判拒答；H2 用统一分类提示词加预定义拒答模式列表；guard 模型以 unsafe 预测为拒答。论文未报告重复次数与统计检验。

    主结果

    H1 拼接双题（Table 2，Acc/RR，%）：

    Swipe horizontally to read the table

    模型防御GPQA 第1题 RRGPQA 第2题 RRMMLU 第1题 RRMMLU 第2题 RR
    Llama-3Base0.08.00.54.0
    Llama-3+StrucQ14.066.06.090.0
    Llama-3+SecAlign4.050.03.534.0
    MistralBase24.016.02.72.3
    Mistral+StrucQ14.034.02.462.2
    Mistral+SecAlign6.074.01.189.6

    （据 Table 2；AIME 列基座准确率均为 0.0，此处省略）

    • 作者称防御模型对第 1 题表现接近正常，但"激进拒答"第 2 题，后缀拒答率常超 50%、最高 90%，而同样题目单独出现时能正确处理。
    • Table 1（单题 vs 拼接整体）同向：如 Llama-3+StrucQ 在 MMLU 上 Acc 从 65.0 降至 32.5、RR 从 2.2 升至 48.0。
    • 例外：Mistral 基座在 GPQA 第 1 题 RR 已达 24.0，高于其 StrucQ 变体的 14.0；论文未解释。

    H2：token 触发偏置（Table 3）

    • 测了什么：基线安全集与触发压力集上的良性拒答率。结果：Llama-3+SecAlign 从 0.00 升至 15.93（+8.85），Mistral+StrucQ 从 11.03 升至 15.04（+12.39）；guard 中 LakeraGuard 从 9.1 升至 58.5（+49.4）、ProtectAIv2 从 13.8 升至 53.9（+40.1）。
    • 作者解读：攻击数据中的高频 token 足以在良性语境激活防御，说明依赖词汇线索而非策略推理；作者称对照的非触发词不引起该上升。
    • 例外：Mistral+SecAlign 触发压力下 RR 为 0.00（∆ -2.65），PromptGuard2 仅 +4.1；FMOPS、Deepset 基线 RR 已高达 65.3、65.9。

    H3 与触发移除消融（Table 4、Table 5）

    • H3 话题泛化：三基准平均 Acc，Llama-3 从 31.6 降至 12.8（StrucQ，-18.8）和 22.4（SecAlign，-3.1）；Mistral 从 23.3 降至 12.1（-11.2）和 10.1（-13.2）。作者称这是分布外良性输入上的"alignment tax"。
    • 触发移除（Table 5）：移除触发词后 guard 的 RR 下降——ProtectAIv2 53.9→23.8（∆ -30.1）、PromptGuard2 11.5→1.70（-9.80）；作者称这为"token 驱动拒答"提供因果证据。
    • 例外：Deepset 移除触发词后几乎不变（87.6→86.7，∆ -0.9），论文未解释。

    其他消融与分析

    • 五题拼接消融（Figure 3，Mistral 基座 vs StrucQ，MMLU 173 组）：响应率基座各位置 0.98–1.00，StrucQ 从第 1 题 0.61 降至第 5 题 0.55；准确率基座约 0.47–0.51，StrucQ 从 0.27 降至 0.23。
    • 输出长度（Figure 2）：作者称 StrucQ 版 Llama-3 在 GPQA、MMLU 上生成长度分布左移，基座呈长尾；作者解读为推理被截断。
    • 定性示例（Figure 4）：同一良性提示词下，原始模型作答而防御模型误拒或答错（位置偏置与触发偏置各一例）。

  5. What are some points that can be further explored?

    The authors state that the analysis is limited to controlled conditions and is only a snapshot of current practice; future directions include counterfactual benign samples and multi-turn diagnostics.

    Limitations and future directions noted by the authors

    • Analysis framework is limited (Section 7 Limitations): The authors describe this as "a focused empirical analysis under a well-defined set of evaluation conditions," not intended to exhaust all possible defenses, and the conclusions should be interpreted within this framework.
    • Snapshot nature (Section 7): The models, benchmarks and defense methods tested are only "a concrete snapshot of current practice"; other designs or deployment scenarios may perform differently; the analysis is complementary rather than exhaustive.
    • Limited metric perspective (Section 7): The evaluation emphasizes interpretable signals such as refusal behavior and task performance; the authors say similar diagnostic principles could be used to explore other perspectives on system behavior.
    • Counterfactual benign samples (Section 5 Future Work): Introducing counterfactual samples that superficially resemble attacks but have benign intent might reduce reliance on lexical and positional cues, but the authors say their impact on the trade-off between attack refusal rate and benign utility is unclear.
    • Interactive and multi-turn settings (Section 5): Extending the diagnosis to interactive or multi-turn scenarios might reveal how early refusals affect downstream behavior in real deployment.
    • Finer-grained behavioral signals (Section 5): Incorporating signals such as reasoning depth and calibration beyond accuracy might more fully characterize the safety–utility trade-off introduced by defense fine-tuning.

    Experimental coverage

    • The generative models tested are two families, Llama-3-8B and Mistral-7B (Appendix B.3), each with three states: Base, +StrucQ, +SecAlign; prompt guards cover five: ProtectAIv2, LakeraGuard, PromptGuard2, Deepset, FMOPS.
    • Benign diagnostic data are GPQA 100 questions, MMLU 1,204 questions, AIME 2024 30 questions; the H2 safety baseline is about 3.9k items, and the trigger stress set has 113 items per level; the five-question ablation is 173 groups (Appendix B.2.2).
    • Defense methods cover only two types of fine-tuning defenses (StrucQ, SecAlign) and external filters; the paper discusses model-level defenses such as Constitutional AI, deliberative alignment, and instruction hierarchy (Section 2.2), but did not run diagnostics on them in the experiments.
    • H3 did not perform an explicit topic split: the Table 4 caption notes "no explicit topic split is applied"; the topic generalization conclusions are based on the domain shift of the three benchmarks relative to the defense training distribution.
    • The paper does not report the number of repetitions per condition, statistical significance tests, or the specific numbers for the H2 control-word experiment (the main text only states that the control does not cause an increase in refusals).

  6. Summarize the main content of the paper

    The authors say mainstream fine-tuning defenses learn three types of surface shortcuts—position, token, and topic—and that high attack refusal rates mask systematic harm to benign inputs.

    A behavioral analysis paper targeting prompt injection defenses: the authors question whether supervised fine-tuning defenses (StrucQ, SecAlign) and external prompt guards truly recognize malicious intent.

    • Problem: Defense training data is dominated by explicit injection patterns, and the authors argue that empirical risk minimization leads the model to learn the rule "refuse whenever surface features appear," so high attack refusal rates on standard benchmarks are treated as misleading evidence of robustness.
    • Method: Three hypotheses are formalized—position bias (the later segment of a concatenated prompt is treated as suspicious by default), token trigger bias (high-frequency attack words trigger refusals in benign contexts), and topic generalization bias (performance on benign tasks in unseen topics degrades); a diagnostic set that preserves benign semantics and varies only a single factor is constructed accordingly, and refusal rates and accuracy are measured on Llama-3-8B, Mistral-7B, and five prompt guards.
    • Result one (position): With two questions concatenated, Llama-3+StrucQ reaches a refusal rate of 90.0% on question 2 of MMLU and only 6.0% on question 1 (Table 2); in the five-question ablation, the response rate decreases monotonically with position (Figure 3).
    • Result two (token): After inserting a single trigger word, the benign refusal rate of LakeraGuard rises from 9.1% to 58.5% (+49.4), and that of ProtectAIv2 rises from 13.8% to 53.9% (Table 3); after removing the trigger word, the refusal rate of ProtectAIv2 drops by 30.1 percentage points (Table 5).
    • Result three (topic): After defense fine-tuning, the average accuracy on three reasoning benchmarks drops by up to 18.8 (Llama-3+StrucQ, Table 4), which the authors call an alignment tax on benign utility.
    • Authors' conclusion: Aggregate refusal metrics are insufficient for evaluating defenses; intent-aware evaluation that preserves benign utility is needed; the authors also note that the analysis is limited to controlled conditions and is a supplementary snapshot rather than an exhaustive treatment of all defenses.
Read originalACL 2026