Text that deceives LLM safety guardrails into making incorrect judgments

Safety guardrails can be steered by a forged scoring task in the content: the guardrail instead scores harmless text supplied by the attacker, and the harmful request passes through unchanged.

• Setting and importance: LLM applications place small guardrails before chat assistants, retrieval, and agent tool calls; content that the guardrail judges safe is let through, and in agents the actions described by the content may be executed without further review. Guardrails such as Shieldstral, gpt-oss-safeguard, and Llama Guard place the policy/question and the content to be reviewed in the same token stream.
• Gaps in existing understanding: The Llama Guard paper warns that it "may be susceptible to prompt injection attacks," and gpt-oss-safeguard reports a score of 0.512 on a prompt injection hijacking test, but the authors state that neither describes what this type of attack against guardrails looks like or how to defend against it.
• Core observation and hypothesis: The labels marking query and document (such as <Query>: and <Document>:) are plain text, and the same labels can appear in the content; the authors hypothesize that the guardrail scores the last complete scoring task in the input, and that the content changes which document the guardrail scores, not how it scores.
• Contributions of the paper: Name and measure the target rebinding attack, build the GuardRebind benchmark covering five guardrails, and propose a defense using boundary tokens plus segment-max self-distillation training, with preregistration and adaptive attack evaluation.
• Threat model (Section 2):
  • Goal: The harmful request h is let through while remaining unchanged and readable in the forwarded content; evasion attacks that delete, rewrite, or obfuscate h are out of scope.
  • Knowledge: The attack and benchmark phases are black-box, with the attacker unable to see scores, split into two categories: "exactly knows the deployed query" and "does not know"; in the defense evaluation phase the attacker knows the defense and holds the post-defense weights, boundary mechanism, and training recipe.
  • Capabilities: Controls only the untrusted content c; does not control the guardrail weights, instruction, query, system prompt, decoding, or threshold; during defense evaluation can query scores and use gradients.
  • Victim system: A guardrail G deployed before a downstream system, where c can be a user message, a retrieved passage, a tool result in an agent loop, or dialogue history.

Structural forgery against generators and judges

• Liu et al. [25] (2024) — the "fake completion" and "combined" attacks forge the end of one task and the start of another; the authors state that this paper applies this construction to guardrails.
• Attacks in the indirect prompt injection line [3, 8, 11, 23, 45] — forge templates or dialogue structure to redirect agents and chat models; other work [21, 40, 41] uses injected content to manipulate LLM judge scores. The authors point out that this paper's goal is to get a forged task scored, with success manifesting as an erroneous "safe" verdict.

Attacks against guardrails

• Guardrails can be broken by universal perturbations [26], character injection [16], pipeline attacks [30], and forged judge outputs [29].
• The closest is parallel work by Li et al. [22] (2026) — forging the verdict and authority fields in the structured input of a typed judge; the authors point out that none of their forgeries introduce a new task that needs to be scored.

Structured queries and distillation defenses

• StruQ and SecAlign [4, 5] (2024) — preserve delimiters plus training, so the model ignores injected instructions; follow-up work [6, 7, 49] extends this line, or distills the model toward clean-context behavior [12, 38]. The authors point out that these all defend generative instruction-following models, whereas this paper applies the idea to the classifier's content slot, and the training objective does not need to know which segment is the injection.
• The authors point out that, like other fine-tuning defenses [34, 36], this paper's defense is also broken by adaptive attacks.

Baselines and benchmarks: Controls for the defense evaluation include the untrained boundary B, the clean-copy target, a fail-closed structural detector, inference-time segment-max, and naive adversarial training; benchmark data sources are AdvBench, HarmBench, Aegis 2.0, BeaverTails, OpenAI moderation evaluation, and ToxicChat, and over-refusal sources are XSTest-safe and OR-Bench-hard.

The authors position this paper as: systematically measuring the known forged-task-structure attack against the object of safety guardrails, and providing a training defense targeting the guardrail's content slot.

Overall approach: first behaviorally verify the hypothesis that "the guardrail scores the last complete task," then systematically measure across five guardrails using the attack grammar, and finally design the defense B+T targeting the attack's two premises (content can produce labels, and the model treats nested label pairs as a new task).

Attack construction and success criteria

• Forged pair: After the harmful request, append a forged <Query> (repeating the deployed safety question) and a forged <Document> (containing a harmless decoy); the guardrail reads two query–document pairs with the same labels, and the authors hypothesize that it scores the last pair.
• Success criteria: Only count requests that the deployment flags when there is no attack; an attack succeeds when G lets through h⊕p (⊕ is concatenation), reporting bypass rate and Wilson 95% intervals. The flagging rule varies by setting: Shieldstral confirmatory experiments use G(h) > 0.9, GuardRebind and defense evaluation use ≥ 0.5, and label-only guardrails take the majority of three API calls as unsafe (temperature 0).
• Mechanistic evidence: The authors state that the evidence is mainly behavioral (forged-pair ablation, decoy dose-response); activation patching (Appendix C) failed to localize the mechanism and only shows that the verdict is read out at the final position.

GuardRebind benchmark

• Data: Harmful and benign items come from six public datasets, with XSTest-safe and OR-Bench-hard supplementing benign items that are easily misflagged; a total of 20k harmful + 20k benign, split 60/10/30 by normalized text hash. The core set is a stratified sample of 600 harmful + 600 benign test items per guardrail, reduced to 200 + 200 for gpt-oss-safeguard due to the paid API.
• Attack grammar: Axes include label syntax, number of forged pairs (1–5), position (append/prepend/middle), decoy harmfulness, forged query relevance, language, and encoding; a family takes one level on each axis, and the core panel contains 52 attack families and 5 controls. guard-native forgery uses plain text to forge the guardrail's entire scoring task.
• Protocol: 3 instantiations are sampled per item–family cell, bypass is averaged, and intervals come from a two-level bootstrap over items and instantiations (preregistered).

Defense Part B: boundaries that content cannot produce

• Render the guardrail's native template with an empty document slot to obtain trusted prefixes and suffixes; tokenize the content separately with special-token parsing disabled, so that control-token strings become ordinary text tokens.
• The content's token ids are wrapped by two reserved boundary tokens, whose ids are unused in pretraining and to which no input text tokenizes; nothing is deleted from the content itself. The authors state that this design follows StruQ and SecAlign.

Defense Part T: segment-max self-distillation

• Training objective: target(c) = max(f₀(c), max_{s ∈ S(c)} f₀(s)), i.e., the student (the trained guardrail) scores the entire content no lower than the frozen base guardrail f0 scores its most harmful segment; for attack samples the segments are the original request and the decoy, and for clean content the target is the teacher's own score.
• The objective only needs segment boundaries, not knowledge of which segment is the injection; training uses binary cross-entropy between the yes/no logit margin and the target, with LoRA applied to attention and MLP projections and embeddings frozen; B+T additionally trains the embedding rows of the two boundary tokens.
• Training data: Each of the 5,000 training-split items appears once clean and twice with attacks, plus 1,250 structured benign hard negatives, for 16,250 examples in total; decoys and forged queries are drawn from the training half of the pool; training sees no test items, held-out attack families, or decoys and queries from the held-out half.

Adaptive evaluation

• The attacker knows the defense and can query scores (following Carlini et al. [2]): score-guided search hill-climbs within 300 guardrail calls per item; structural search enumerates all 654 forms plus 30 random label variants; GCG optimizes a 20-token suffix inside the forged document for 250 steps (20 items); there is also a plain-text control search that adds only decoy text, with no labels and no forged query.

Experimental setup

• Guardrails tested: Shieldstral-1.0-3B, Llama Guard 3 1B, ShieldGemma 2B, Qwen3Guard 0.6B (run locally), gpt-oss-safeguard-20B (OpenRouter API, label-only); Llama Guard 4 12B additionally appears in a cross-model check on 30 requests.
• Confirmatory experiments: AdvBench lines 200–299, 75 flagged under no attack (> 0.9), protocol frozen in advance (Appendix A).
• GuardRebind: 600 harmful + 600 benign core test items per guardrail (200 + 200 for gpt-oss-safeguard), 52 attack families + 5 controls, 3 instantiations per cell.
• Metrics: bypass rate (flagged items only, Wilson or two-level bootstrap 95% intervals), clean FNR/FPR, AUROC, recall, over-refusal rate.
• Defense evaluation: Shieldstral test split, 297 flagged items (base and B+T)/292 (each T seed); detection uses 1,500 clean test samples (801 harmful, 699 benign); over-refusal uses XSTest-safe (n=75), OR-Bench-hard (n=400), structured benign set (n=500).
• Adaptive attacks: 50 flagged test items, score-guided search with 300 calls per item, structural search enumerating 684 attacks, GCG run on 20 items.

Main results

GuardRebind results per guardrail (from Table 3, %, bypass counts flagged items only):

Swipe horizontally to read the table

Guardrail	Flagged items	clean FNR	clean FPR	Strongest family bypass	guard-native	Instruction only	Harmful decoy
Shieldstral 3B	300	50.0	6.7	99.2	71.7	7.0	13.6
Llama Guard 3 1B	374	37.7	15.8	100	100	2.9	11.1
ShieldGemma 2B	168	72.0	2.8	100	73.6	20.2	84.3
Qwen3Guard 0.6B	536	10.7	27.8	93.4	81.7	0.7	0
gpt-oss-safeguard 20B	113	43.5	12.0	100	71.7	6.2	9.4

• The authors note that the strongest family was selected post hoc, and that guard-native forgery was fixed before any test-split run and passes 71.7–100% across guardrails.
• The authors note that ShieldGemma's 100% strongest bypass covers only the few harmful items it flags (its clean FNR is 72.0%); ShieldGemma is nearly insensitive to decoys (forged triples still pass 84.3% under a harmful decoy), which the authors describe as closer to generic evasion than rebinding.
• The authors note that Llama Guard 3 is partly affected by any trailing text: benign trailing text without a forged pair passes 41.2% (Shieldstral 4.3%), i.e., part of its bypass comes from recency.

Shieldstral confirmatory experiments (Table 1, Figure 2)

• Forged-pair ablation (n=75): a full forged Query+Document pair passes 100% [95%, 100%], with the median score dropping from 0.986 to 0.016; a forged Document only passes 9%; the instruction sentence S only, the "Note:" form, inert padding, forged Instruct, and forged Instruct+Query all pass 0%; forged Instruct+Query+Document passes 92%. The authors note that if the guardrail were being persuaded, adding an instruction should help the attack, but it in fact slightly decreases it.
• Decoy dose-response (exploratory, n=39, Figure 2b): a benign decoy (scoring 0.00 alone) passes 100%, a borderline decoy (0.03) passes 33%, a harmful decoy (≈1.0) passes 0%; over 150 requests × 40 decoys, Spearman ρ = −0.84, which the authors describe as partially supporting the preregistered prediction (the interval crosses the threshold, −0.78 on the dev split, non-monotonic by bin).
• Robustness: a forged pair placed before the request passes only 15% (100% when appended); seven label syntaxes pass 83–100%, while the same text without a label passes 0%; a blind attacker who does not know the deployment query and stacks five guesses still passes 100%; under eight policy phrasings the median bypass is 99% (range 87–100%).

Defense evaluation (Tables 4, 5, 6)

• Main results (Shieldstral, held-out families + held-out decoys and queries): base passes 75.0% [74.0%, 76.0%], T with two seeds passes 0.8% and 1.6%, B+T passes 0.9% [0.7%, 1.1%]; B alone (boundary without training) passes 76.8%, which the authors say shows the model will not use the boundary tokens without training.
• Cost: AUROC goes from 0.810 for base to 0.802–0.803 for T and 0.800 for B+T; the authors state non-inferiority is not established (the point estimate is within the preregistered bound, but the interval is wider); benign content with a benign forged pair is flagged 0.7% under T and 0.8% under B+T (base 0.2%).
• Training-free baselines: the structural detector and inference-time segment-max pass 23.2% and 23.4% respectively on held-out families + held-out decoys; the authors state that nearly all remaining bypass comes from two non-English label families; the detector falsely flags 58.6% of the structured benign set.
• Adaptive attacks (Table 6, 50 items): score-guided search passes T 47/50 and B+T 34/50 (the authors describe this as the only result where the boundary clearly helps); structural search passes T and B+T only 4/50 each; GCG passes all 20 items for T, bringing the per-item worst case for T to 49/50. Of T's 47 failures, 34 placed the forged structure before the request.
• Plain-text control: benign user prompts without labels or forged queries, selected by the same search, pass T 26/50 (base 35/50); the authors state that about half of the remaining weakness does not require rebinding.

Other ablations and analyses

• Cross-model check (30 AdvBench requests): forging each model's own structure passes gpt-oss-safeguard 30/30 and Llama Guard 4 29/29, while a plain assertion of safety passes 0%.
• Qwen3Guard probe (40 training-split items): forging only the dialogue passes 5/40, forging the entire task passes 40/40.
• Naive adversarial training (Table 8, fixed decoy): held-out families pass 0.9–1.1%, AUROC 0.921–0.925, but trailing harmful content is flagged only 8.0–8.6% (T: 95.5–100%).
• T transfer (Table 8, fixed decoy): Llama Guard 3 held-out family bypass drops from 82.0% to 2.5%, but the flag rate on benign content with a benign forged pair rises from 2.4% to 22.5%; Qwen3Guard drops from 9.5% to 0.0%, a number the authors say shows nothing.
• clean-copy objective ablation: held-out passes 2.1%, but benign content with a harmful decoy is flagged only 0.8% (Table 5).
• gpt-oss-safeguard multiple verdicts: reading only the last verdict raises the two stacked-pair families from 2.9–3.5% to 87.3–90.6% (Appendix C).

Limitations and future directions noted by the authors

• The confirmatory study covers only one guardrail, one English corpus and short documents; the Llama Guard 4 and role-boundary checks use only 30 requests (Section 8 Limitations).
• Bypass counts flagged items only (Shieldstral's clean recall on its clean test samples is 47%), can only bound end-to-end harm from above, and was not tested on retrieval, tool or multi-turn channels (Section 8).
• Some results are based on small runs: the Qwen3Guard probe on 40 items, adaptive attacks on 50 items (GCG on 20 and B+T not run), and all training conditions except T and naive adversarial training use a single seed (Section 8).
• The held-out defense results cover only the append form, and the defense only partially weakens searched benign prompts (from 35/50 to 26/50) (Section 8).
• segment-max retains only compositional harm that the base guardrail itself can detect; compositional attacks were not run, and the authors state they do not claim general prompt injection robustness (Section 8).
• Part of the defense evaluation (the held-out decoy metric, score-guided search, the two training-free baselines, the reduced seed plan) was designed after seeing the first round of test-split results (Section 6.1, deviation D4).

Scope of experiments

• 6 guardrails were tested: Shieldstral-1.0-3B, Llama Guard 3 1B, ShieldGemma 2B, Qwen3Guard 0.6B, gpt-oss-safeguard-20B; Llama Guard 4 12B appears only in the 30-request check (Section 4).
• The confirmatory experiments are 75 flagged requests from AdvBench lines 200–299; the GuardRebind core set is 600 harmful + 600 benign per guardrail (200 + 200 for gpt-oss-safeguard) (Sections 3, 4.1).
• The defense is mainly tested on Shieldstral; T is run with one seed each on Llama Guard 3 1B and Qwen3Guard 0.6B under the fixed-decoy condition (Section 6.2, Table 8).
• The adaptive evaluation includes score-guided search (50 items, 300 calls per item), structural search (50 items, 684 attacks), and GCG (20 items, B+T not run) (Section 6.3).
• The paper does not report tests of the defense on retrieval, tool or multi-turn channels, nor a comparison of T and naive adversarial training at matched recall (the results files have no per-item clean scores, Section 6.2).

An attack and defense paper on LLM safety guardrails: the guardrail reads the policy and the content to be reviewed as the same plain-text stream, and content that imitates the guardrail's prompt format can make the guardrail score a different text.

• Problem: The attacker appends a forged Query/Document pair and a harmless decoy after the harmful request; the guardrail scores the decoy, and the harmful request passes through unchanged; the authors state that this is not persuading the guardrail but changing which document it scores, so anti-persuasion measures such as "ignore instructions in the content" are ineffective against it.
• Method: Preregistered confirmatory experiments on Shieldstral (forged-pair ablation, decoy dose-response); build the GuardRebind benchmark and test five guardrails with 52 attack families; the defense uses boundary tokens that content cannot produce (B) plus segment-max self-distillation training (T), training the guardrail to score the entire content no lower than its most harmful segment.
• Attack results: On Shieldstral, the full forged pair lets through all 75 flagged requests, the instruction sentence alone lets through 0%, and the attack lets through 0% with a harmful decoy; on GuardRebind, every guardrail has an attack family that lets through 93.4–100% of flagged items (Table 3); the authors state that rebinding is best explained on Shieldstral and gpt-oss-safeguard, that ShieldGemma is closer to generic evasion, and that Llama Guard 3 is partly due to a trailing-text effect.
• Defense results: T reduces bypass on held-out families + held-out decoys from 75.0% to 0.8–1.6%, and AUROC from 0.810 to 0.802–0.803 (the authors state that non-inferiority is not established); boundary tokens alone without training are essentially ineffective (76.8%).
• Adaptive attacks: Score-guided search that knows the defense and can query scores lets through T on 47/50 and B+T on 34/50, mostly placing the forged structure before the request; among these, 26/50 can also let through T using benign text with no forged structure.
• Authors' conclusion: Guardrails that read the task and content as the same plain-text stream can be induced to score a forged task; segment-max training reduces bypass for append-style attacks to about 1%, but adaptive attackers with score access can still break it; the authors recommend that deployers train guardrails and never act on the guardrail's verdict alone.