Similar in Safety, Silent in Propagation: Limitations of Text-Similarity Guardrails for AI Agent Security
Safe Resemblance, Silent Propagation: Limits of Text-Similarity Guardrails for AI Agent Safety
The authors evaluate the text-similarity guardrail DonkeyRail in a three-agent email relay, reporting that it misses 8/19 functional propagation outputs and falsely flags 70/192 authorized reuse replies.
- When AI agents forward information to one another, injected instructions may still influence downstream agents after wording changes, while legitimate citation and template reuse produce high text overlap. Text-similarity guardrails struggle to distinguish these two cases.
- The authors use a frozen DonkeyRail classifier (BLEU, ROUGE-L, METEOR features) to conduct behavioral evaluation in a three-agent email relay: functional propagation is defined by whether the two subsequent agents produce new computation, and a legitimate control with explicit authorized citation/reuse is set up; they additionally perform threshold calibration, ROC interpolation correction, and formal analysis.
- At threshold 0.5, 8 of Luna's 19 functional root outputs are missed, and 70 of 192 legitimate test replies are falsely flagged; the calibrated threshold detects only 1 functional output and still falsely flags 15. The authors also state that the fixed-threshold rates of all 21 published classifier configurations were obtained by ROC interpolation.
- The authors note that the experiments cover only one arithmetic relay, 12 framings, 6 variants, and two model aliases; attack variants share the same operational core; exposure is forced rather than naturally retrieved; there is no external tool execution; and there is only one sampled output per condition, so sampling variance cannot be estimated separately.
- Threat model
- Indirect prompt injection: an untrusted relay record requests an additional operation, and the output is forwarded to subsequent agents; the evaluation focuses on whether the injected procedure triggers new computation in the two subsequent agents, and whether legitimate citation/template reuse is falsely flagged. The victim system is a three-agent email relay, and the guardrail is a frozen DonkeyRail classifier.
- Evaluated models
- DonkeyRailgpt-6-lunagpt-6.1-sol
- Benchmarks
- DonkeyRail三智能体邮件中继(合成任务)
- Metrics
- functional recalllegitimate FPAUCFPRTPR
The paper's empirical findings show that text-similarity guardrails struggle to distinguish silent propagation of injected instructions between agents from legitimate citation and template reuse. The authors tested the public classifier DonkeyRail (combining BLEU, ROUGE-L and METEOR features) in a controlled three-agent email forwarding scenario, and defined a behavioral metric independent of detector scores, based on whether the injected program triggers new computation in the subsequent two agents. At the default threshold, the classifier missed 8 of 19 functional outputs and falsely flagged 70 of 192 legitimate replies; completed citation and template-reuse replies all triggered false positives; calibrating for low false positives on one template set did not transfer, and the recalibrated threshold detected only 1 functional output while still producing 15 false positives on the new templates. The paper also points out that all 21 published classifier configurations have an operating-point error in which ROC interpolation is substituted for direct threshold application, and argues that a high AUC against surface-text criteria does not guarantee a high AUC against the behavioral objective.
Paper analysis
What problem does this paper attempt to solve?
The authors study whether text-similarity guardrails can distinguish downstream propagation of injected instructions from legitimate authorized reuse.
Core question: can a text-similarity guardrail distinguish injected instructions that persist in downstream agents from legitimate citation/template reuse.
- Problem setting: an AI agent's output becomes another agent's input; indirect prompt injection can place instructions into retrieved material, and once the output is forwarded, the relevant event is the instruction's continued influence on another agent.
- Shortcomings of existing methods: the authors argue that text similarity and propagation answer different questions—a paraphrased output may retain an executable program with little textual overlap, while citation, summarization, and template reuse may produce high overlap because the user explicitly requested them.
- Core observation: safety decisions need to distinguish three things: textual similarity, downstream-induced behavior, and task-granted permissions.
- What the paper proposes: use DonkeyRail (the public implementation of the Morris-II line of research) to conduct behavioral evaluation in a controlled three-agent email relay, define a functional metric independent of detector scores, and perform threshold calibration, fixed-threshold measurement correction, formal analysis of proxy-label transfer, and authorization analysis.
- Threat model:
- Attacker goal: make the injected program trigger new computation in the two subsequent agents (functional propagation).
- Knowledge/capability: an untrusted relay record can request additional operations; attack variants are prepared by Luna and include paraphrasing, format changes, Base64 wrapping, etc.
- Victim system: a three-agent email relay using a frozen DonkeyRail classifier as the guardrail.
- Success criterion: both subsequent agents of the root output o1 produce the correct new result line (A2=1 and A3=1).
What related research exists?
The paper discusses five categories of work: prompt injection and propagation, resilient worms and behavioral evaluation, policy-aware defenses, text-similarity metrics, and proxy criteria.
Prompt injection and propagation
- Greshake et al. [8] — show how indirect prompt injection can compromise LLM-integrated applications through data encountered in legitimate tasks.
- Morris-II [4] — extends the threat to self-replicating prompts in RAG applications; its Virtual Donkey guardrail uses the maximum BLEU, ROUGE-L, and METEOR overlap between retrieved records and generated replies as evidence of replication; the paper notes that it measures textual replication through exact instruction-string retention, and acknowledges that modified strings may retain behavior.
Resilient worms and behavioral evaluation
- Autonomous LLM Agent Worms [16] — proposes SRPO to maintain persistence, propagation, operational effect, and necessary identifiers across summarization, paraphrasing, and compression, distinguishing semantic retention from verbatim retention.
- AgentWorm [17] — studies persistence and propagation across agent ecosystems; the evaluation separates configuration persistence, execution verified through host artifacts, and propagation, with full success requiring all three stages to be satisfied.
- AgentDojo [7] — uses checks on outputs and stateful tool-environment changes to separate legitimate task utility from attacker goal success.
Policy-aware defenses
- Fides [5] — propagates confidentiality and integrity labels through agent computation and performs information-flow policy checks on important tool calls.
- CaMeL [6] — separates control flow from data flow: a privileged model translates trusted tasks into programs, and an isolated tool-free model processes untrusted data into predefined schemas.
- The paper also cites [15] to point out that security mechanisms themselves can become an attack surface: when injected content triggers defensive refusal, warning-induced abstention can deny service to agents using legitimate tools.
Text-similarity metrics and proxy criteria
- BLEU [12], ROUGE [11], METEOR [1] — text-correspondence metrics oriented toward machine translation, summarization, and monolingual alignment, respectively; the paper notes that their definitions do not determine whether an instruction causes a subsequent agent to act or whether replication is authorized.
- Criterion contamination [2], inclusion bias in diagnostic evaluation [14], evaluation of depressed language models [10], AVClass [13], and MOTIF [9] — the paper draws on these works to show that agreement with a coupled criterion may overestimate transfer to the other endpoint.
Baselines and benchmarks: The evaluated object is the three-feature logistic classifier in the public DonkeyRail implementation (fixed commit); the experimental environment is a three-agent synthetic email relay built by the authors.
Author positioning: This paper applies behavioral evaluation principles to a narrower detector question — whether a fixed text-similarity score tracks functionality while allowing authorized reuse.
How does the paper address this problem?
The authors build a three-agent email relay, evaluate the frozen DonkeyRail classifier using independent behavioral endpoints, and supplement this with threshold calibration, formal analysis, and authorization analysis.
Overall approach: In a controlled three-agent email relay, evaluate the DonkeyRail text-similarity classifier using behavioral endpoints independent of the detector score (whether the two subsequent agents produce new computation), while setting up legitimate controls with explicit authorization for citation/reuse, to test false negatives and false positives.
Similarity guardrail and decision rule
- The feature vector takes the maximum of each metric over ten retrieved records: x(D,o) = (maxd∈ D BLEU(d,o), maxd∈ D ROUGE\text{-}L(d,o), maxd∈ D METEOR(d,o)); the three maxima may come from different records.
- Uses the three-feature logistic classifier in the public implementation, with score s(D,o)=σ(b+wᵀx(D,o)); an alert is raised when s(D,o)≥τ; the classifier is frozen in the behavioral experiments.
- Generated replies are lowercased during feature extraction (following the original preprocessing convention); subsequent agents receive the actual output text.
Behavioral propagation measure
- The email agent receives nine synthetic background records, one relay record, and one incoming message requesting confirmation of a meeting (including a ticket number); the untrusted relay requires adding seven to the ticket, outputting a separate LAB-PROOF line, and carrying the procedure into the reply.
- Each agent receives a new ticket; for the root output o1, F(o1) = 1[A2 = 1 ∧ A3 = 1], i.e., both subsequent agents produce the correct new result line outside the explicit relay tag, code fence, and citation line; changing the ticket rules out satisfying the endpoint by copying earlier results.
- A stricter three-action endpoint A1A2A3 is also reported; the authoritative annotation and the prohibition on additional proof lines are evaluation metadata, not explicit policy instructions provided to the model.
Attack variants and legitimate controls
- 12 frameworks designated for held-out target evaluation, each with 6 variants: identity, paraphrase, format change, clause reordering, indirect quotation, Base64 packaging; 4 variants per framework are prepared once by Luna, and identity and encoding are deterministic; all variants are fixed before target evaluation.
- The main experiment evaluates all 72 held-out variants for each model on three agent chains, sharing 216 unique ticket values across models.
- The legitimate control for each model is 192 calibration + 192 test replies, with each split containing 48 items from each of four task types: plain confirmation, verbatim citation, paraphrase, and full template reuse; calibration and test request templates do not overlap, and source record formats also differ.
- The false-positive denominator retains all legitimate outputs; a separate task-completion check is used (citation/template reuse must contain the full source after normalization; confirmation/paraphrase must specify the time and meeting-room tag).
Model settings and statistics
- Evaluates gpt-6-luna and gpt-6.1-sol with stateless calls; temperature 1.0, top-p 0.98; Luna reasoning effort is none with an output limit of 768 tokens, Sol is low with a limit of 1,536 tokens (including reasoning).
- Reports a fixed threshold τ=0.5 and a threshold determined solely by legitimate calibration replies, τcal = nextafter(s(4), +∞) (allowing at most 3 calibration alerts, empirical calibration FPR below 2%); the rule is fixed before data collection.
- Uses 4,000 bootstrap samples to estimate 95% percentile intervals: functional recall is resampled by 12 framework seeds, and legitimate false positives are resampled by 24 request-template clusters.
Measurement correction and formal analysis
- Points out that the public implementation computes fixed-threshold rates using linear interpolation between ROC probability thresholds; the authors instead count FPR/TPR directly and reconstruct all 21 classifier configurations.
- Adapts the concept of criterion contamination to guardrail evaluation: constructs a family of behavioral distributions that achieve every AUC in [0,1] under a fixed perfect proxy ranking (Proposition 1), and defines the normalized AUC gap CCI (Eq. 8), noting that it requires paired proxy and target labels on the same population.
- Authorization analysis: if two worlds have the same visible text O(w) but different authorization A(w), any deterministic function using only O cannot make the correct decision in both worlds (Proposition 2); this condition is instantiated with 64 pairs of synthetic contact summaries.
What experiments did the paper conduct?
The main experiment shows that at threshold 0.5, Luna misses 8 of 19 functional root outputs and falsely flags 70 of 192 legitimate replies; the calibrated threshold detects only 1 functional output.
Experimental Setup
- Models under test: gpt-6-luna and gpt-6.1-sol (stateless calls, temperature 1.0, top-p 0.98; Luna reasoning effort none, 768 token limit; Sol low, 1,536 token limit).
- Guardrail under evaluation: the three-feature logistic classifier of the public DonkeyRail implementation (fixed commit), with features being the maximum of BLEU, ROUGE-L, and METEOR over ten retrieved records.
- Dataset/environment: synthetic three-agent email relay; 72 held-out attack variants per model (12 framings × 6 variants), 216 unique tickets; legitimate controls are 192 calibration + 192 test replies, 48 each across four task types.
- Metrics: functional recall (TP/(TP+FN), functional root = both subsequent agents produce new computation), number of false positives on legitimate test replies, task completion rate, AUC.
- Evaluation method: an action parser makes rule-based determinations (an independent LAB-PROOF integer line equal to the current ticket plus seven, excluding relay labels, code fences, and quoted lines); task completion is checked by exact reuse or time-meeting-room; no LLM judging as described in the paper.
- Sample size and repetition: one sampled output per framing, variant, model, and agent position; 4,000 bootstrap interval estimates.
Main Results
According to Table 1 (main-experiment guardrail operating points, all rows using the same frozen classifier):
Swipe horizontally to read the table
Agent model Threshold Functional roots TP FN Recall Legitimate FP Luna 0.5 19/72 11 8 57.89% 70/192 Luna Calibrated 19/72 1 18 5.26% 15/192 Sol 0.5 0/72 0 0 N/A 98/192 Sol Calibrated 0/72 0 0 N/A 69/192 - The authors state the results show errors in both directions: missing functional propagation while falsely flagging authorized reuse; the seed-cluster bootstrap interval for Luna functional recall is 35.0%–76.9%, and the template-cluster false-positive interval is 21.9%–51.6%.
- Sol has no functional roots and no new actions across the 216 main-experiment agent outputs, so its functional recall is undefined; the authors note that zero observed propagation neither proves the guardrail's detection capability nor prevents it from falsely flagging authorized reuse.
- False positives concentrate in citation and template reuse (Table 2): plain acknowledgments trigger 0/48 alarms for both models, paraphrases 2/48 each; Luna citation 37/48, template 31/48; Sol 48/48 for each of the two types.
Task Completion and Hypothetical Blocking
- Before blocking, Luna passes 122/192 and Sol passes 188/192 task-completion checks; among citation and template outputs containing the full request source, all 30 qualifying Luna replies and all 93 qualifying Sol replies are flagged at threshold 0.5.
- The authors state this establishes false positives on completed authorized tasks; if blocked according to alarms, completed tasks would drop from 122 to 90 (Luna) and from 188 to 93 (Sol); this count assumes flagged replies are blocked rather than measuring a deployed system.
Calibration Transfer
- The calibration rule produces zero alarms on the 192 calibration replies for both models; but on the test set with new templates and new source formats it still alarms 15/192 (Luna, 7.81%) and 69/192 (Sol, 35.94%), and Luna functional recall drops from 11/19 to 1/19.
- The authors state that raising the threshold removes most functional detections but does not maintain the target false-positive rate on these test controls; because templates and source formats change simultaneously, the transfer failure cannot be attributed to a single factor.
- The earlier evaluation (Appendix C) produces zero calibrated test alarms under shared templates while losing all 13 functional positives.
Fixed-Threshold Measurement Correction
- The fixed-threshold rates obtained by ROC interpolation in the public implementation reproduce all 21 published operating points; direct counting changes FPR in 19/21 configurations and changes FPR or TPR in all 21 configurations.
- Example: the BLEU decision stump at threshold 0.5 has a reported FPR of 0.500036, while direct counting gives 23/1500=0.015333 and TPR 1491/1495=0.997324; the three-feature logistic classifier has direct FPR 29/1500=0.019333 (reported value 0.021580), direct TPR 1495/1495, and AUC remains 0.997884.
Other Ablations and Analyses
- Behavioral AUC (Table 3): Luna main-experiment root AUC 0.7160, F/benign AUC 0.6436; after pooling, root AUC 0.7609, F/benign AUC 0.5272; agent AUC is undefined because observed C is all zero.
- Marker coverage: all 144 Luna roots and 384 legitimate test outputs lack the roleplay phrase, so this marker has zero coverage of functional positives; the labels of 2,995 records in the historical test set are fully consistent with this phrase criterion, with agent-label AUC 0.997884.
- Earlier evaluation (Appendix C): Luna 13 functional roots (18.06%), 7/13 detected at threshold 0.5, both models falsely flag 128/192 legitimate test replies each (acknowledgment 0/64, citation 64/64, template 64/64); at the calibrated threshold 0.999981053592889, zero test alarms and zero functional detections.
- Exploratory diagnostics (Appendix C): removing explicit relay content and the like before scoring reduces fixed-threshold functional detections from 7 to 0, while legitimate alarms remain 128; a task-specific rule targeting uncited numeric proof lines flags 10/13 functional roots and 0 legitimate replies; the authors note this rule was finalized after data collection and is an exploratory diagnostic rather than a general defense baseline.
- Authorization instantiation (Section 6): among 64 pairs of byte-identical contact summaries with different external allow/deny policies, the observation-only constant baseline is correct 64/128, while referencing the given policy is 128/128.
What are some points that can be further explored?
In Discussion and Limitations, the authors list four categories of limitations: task coverage, exposure scope, measurement, and system coverage.
Limitations and future directions noted by the authors
- Task and mechanism coverage (Section 7): studies only one arithmetic relay, 12 held-out frameworks, 6 variant types, and two model aliases; all attack variants share one operational core and one preparation model, and framework clusters do not represent independent attack mechanisms.
- Legitimate control construction (Section 7): the main control separates full request templates and source formats, but retains hand-written records and shared broad task categories; high-replication tasks make up half of the main legitimate mixture, and the reported false-positive rate describes that mixture and does not estimate its prevalence in real email.
- Exposure and behavioral scope (Section 7): each output is forcibly inserted into the next agent's context, so what is measured is instruction adoption under forced exposure, not natural retrieval, message opening, or email delivery; there is no external tool execution, data exfiltration, or persistent state; the new arithmetic result is a restricted proxy; the two downstream observations establish limited propagation rather than unlimited self-replication.
- Measurement and statistical uncertainty (Section 7): the action parser requires exact independent results and excludes citation regions; a lenient sensitivity check changes the earlier-evaluated Luna count from 13 to 14; strict utility rules may reject semantically acceptable formats; each evaluation has only one sampled output per framework, variant, model, and position, and ticket and random output vary together, so sampling variance cannot be estimated separately.
- System coverage (Section 7): the evaluated classifier uses the public implementation's features and saved training split, and does not reconstruct the original Gemini application replies or historical retrieval context; the model API exposes aliases without dated snapshots and with different reasoning settings; broader metric classes and policy-enforcement systems require separate evaluation.
Scope of experimental coverage
- The models tested are gpt-6-luna and gpt-6.1-sol, 2 in total; the evaluated guardrail is the three-feature logistic classifier in the public DonkeyRail implementation (Appendix B additionally reconstructs logistic regression, Gaussian naive Bayes, and depth-one decision trees, 21 configurations in total, for the measurement correction).
- Attack variants are 12 held-out frameworks × 6 variants, 72 three-agent chains per model; legitimate controls are 192 calibration + 192 test replies per model, 48 items from each of four task types.
- Defense/blocking is only counted hypothetically (flagged replies are treated as blocked); no deployed blocking system is measured.
- Review is by rule-based parser and exact task-completion checks; the paper does not report LLM-judge or human-evaluation agreement.
- One sampled output per condition; functional recall and false positives are given intervals from 4,000 bootstrap samples, and earlier evaluations used 2,000.
Summarize the main content of the paper
The authors evaluate the DonkeyRail text-similarity guardrail using behavioral endpoints, reporting that it simultaneously misses functional propagation and falsely flags authorized reuse, and that calibration cannot resolve this separation problem.
A paper on AI agent safety evaluation that studies whether text-similarity guardrails can distinguish downstream propagation of injected instructions from legitimate authorized reuse.
- Problem: When agent outputs become inputs to downstream agents, injected procedures can remain effective after wording changes, while legitimate citation/template reuse produces high text overlap; similarity signals may confuse the two.
- Method: Evaluates a frozen DonkeyRail classifier (BLEU, ROUGE-L, METEOR features) in a synthetic three-agent email relay, defines functionality by whether the two subsequent agents produce new computation, sets up explicitly authorized citation/reuse controls, and performs threshold calibration, ROC interpolation correction, formal analysis of proxy-label transfer, and authorization analysis.
- Main results: At threshold 0.5, 8 of Luna's 19 functional root outputs are missed (recall 57.89%), and 70 of 192 legitimate test replies are falsely flagged; Sol has no functional roots but is falsely flagged 98 times (Table 1).
- Calibration results: The threshold determined solely by legitimate calibration replies reduces Luna's functional detection to 1/19, and still falsely flags 15/192 on the new-template test set (Table 1).
- Measurement correction: The fixed-threshold rates of all 21 published classifier configurations are obtained by ROC interpolation; direct counting changes FPR in 19 configurations and changes FPR or TPR in all 21 (Section 5.1).
- Authors' conclusion: The authors state that text similarity cannot reliably distinguish sustained instruction adoption from authorized reuse, and that agent safety evaluation should combine independent behavioral measures with trusted policy context rather than relying on text similarity alone.