On-task but off-target: evaluating the robustness of large language models and agents to near-target prompt injection

1. 01What problem does this paper attempt to solve?

   The authors state that existing prompt injection evaluations only test targets that obviously deviate from the task, and will miss near-target attacks that retain a legitimate appearance.

   Core problem: existing prompt injection benchmarks only test attack targets that obviously abandon the user task, and may miss "near-target" attacks that retain most of the appearance of a legitimate answer but introduce erroneous or harmful changes, making systems appear robust when they are not.

   • Scenario and motivation: when LLMs and agents process retrieved documents, external messages, and tool outputs, injected instructions within them may manipulate answers, leak privacy, or induce unauthorized actions. The authors argue that systems may pass tests by refusing attacks that obviously deviate from the task, yet still accept task-relevant erroneous/harmful changes.
   • Shortcomings of existing evaluations: the authors state that searching for more injection instructions against a selected target can only show whether that target can be induced, not whether other targets are easier to induce; evaluations that test only distant targets make a flawed boundary look as effective as an ideal boundary (Figure 2).
   • Core observation: an attack target can retain most of the content of a legitimate answer, changing only key facts (e.g., changing an announcement time from 10 o'clock to 11 o'clock) or attaching harmful behavior (e.g., leaking a private phone number), which the authors call near-target prompt injection (Figure 1).
   • What the paper proposes: NTBench (Near-Target Benchmark), which uses legitimate reference results as anchors to generate target-instruction pairs for the incorrect and harmful tracks, combines LLM checks with manual verification, and then evaluates on 7 LLMs, 10 language datasets, and 2 agent benchmarks.
   • Threat model:
     • Goal: induce the system to produce task-relevant but erroneous or harmful answers/tool calls, rather than obviously abandoning the task.
     • Knowledge: black-box with respect to the victim system, with no parameters, gradients, or internal activations; the attack generator can obtain fields such as the user request, relevant context, and reference result provided by the constructor.
     • Capability: addition-only; it can only insert instructions at the specified injection position, and cannot edit, delete, replace, or reorder the original content, nor modify trusted instructions.
     • Victim systems: LLMs processing untrusted content, and agents encountering injected content during tool use.

   Collapse this answer

2. 02What related research exists?

   Related work falls into three categories: attack methods, evaluation benchmarks, and defenses; the authors state that NTBench complements existing benchmarks from the perspective of target selection.

   Prompt injection attacks

   • Early and manual methods: Greshake et al. (2023) demonstrate indirect prompt injection embedded in external content; Perez and Ribeiro (2022) summarize manual attack templates.
   • Automated attacks: Neural Exec (2024) uses white-box gradient search for execution triggers; GCG and Tree of Attacks with Pruning are adapted to AgentDojo (Hofer et al., 2026); AutoInject (2026) uses reinforcement learning to optimize adversarial suffixes, and PISmith (2026) trains an injection generator against defenses; Zhan et al. (2025) study adaptive attacks against agents and defenses. The authors note that these methods optimize injections for a selected target and do not themselves test other outcomes.

   Prompt injection benchmarks

   • Language tasks: BIPIA (2025) evaluates indirect injection with task-relevant attacks; SEP (2025) measures instruction-data separation.
   • Agent tasks: InjecAgent (2024) and AgentDojo (2024) evaluate malicious goals in tool-use workflows; Agent Security Bench (2025) expands coverage of agent components and attack surfaces; AgentDyn (2026) studies dynamic tasks containing useful third-party instructions.
   • Platform and target design: PIArena (2026) supports adaptive attack generation and studies task-aligned knowledge poisoning; AutoDojo (2026) generates attacks through iterative black-box optimization; StakeBench (2026) links attack goals to stakeholder harm and user intent.

   Prompt injection defenses

   • Detection and filtering: DataSentinel (2025) and PromptArmor (2025) detect or remove malicious content before it reaches the task model.
   • Model-level defenses: instruction hierarchy training (2024), StruQ (2025), SecAlign (2025) and Meta-SecAlign (2025), on-policy distillation in SecOPD (2026), and representation-level privileged embeddings in ISE (2025) and AIR (2025).
   • Agent harness: IsolateGPT (2024), CaMeL (2026), and Progent (2025) perform isolation, information flow control, and permission policy checks at execution time; EvoSafeHarness (2026) automatically evolves harness policies and execution logic.

   Relationship to LocalAlign: The authors state that LocalAlign (2026) uses near-target samples as preference pairs to improve defense generalization and is a training method; NTBench provides an evaluation perspective and benchmark, testing whether systems that resist baseline attacks remain susceptible to related incorrect/harmful outcomes.

   Baselines and benchmarks: The fixed-target baseline is taken from 1,700 original injections in PIArena and the injection tasks of the AgentDojo/AgentDyn suites; the evaluation benchmarks are the 10 language datasets of PIArena, AgentDojo v1.2.2, and AgentDyn. Authors' positioning: NTBench complements existing benchmarks and attack optimizers from the perspective of attack target selection, and near-target goals can expand the range of tested outcomes within existing task environments.

   Collapse this answer

3. 03How does the paper address this problem?

   NTBench uses legitimate reference results as anchors to generate near-target attack pairs, injects them after dual verification, and judges success by target-specific checks.

   Overall approach: anchored on legitimate reference results, use an LLM to generate target-instruction pairs that retain most of the legitimate content but introduce errors or harmful deviations; after verifying that the target indeed constitutes a violation, inject the instruction into untrusted content and check whether the victim system is induced. NTBench has two tracks, incorrect-target and harmful-target, and reports success rates separately.

   Problem formalization

   • The system input is represented as x′ = (t, u′), where t is the trusted user request and system instructions, and u' is the untrusted content after the attack instruction is inserted at a specified position, with the original content and t kept unchanged.
   • The near-target set is defined as T_near(τ, y_ref, ϵ) = N_τ,ϵ(y_ref) ∩ D_τ: the set of targets whose dissimilarity from the reference result y_ref does not exceed ϵ and which belong to the erroneous or harmful deviation D_τ; paraphrases and other legitimate answers are excluded.
   • Attack success is defined as: the injected instruction causes the system to realize the violation condition specified by target a; it does not require reproducing the remaining benign details of a, nor does it require the observed result to still fall within the neighborhood.

   Target and instruction generation

   • Reference result: for language tasks, generated by the reference model f_ref on the input without injection; for agent tasks, the next action or answer after the injection point in the source benchmark's ground-truth trajectory.
   • Generation method: the generator LLM, following track-specific prompts, samples candidate target-instruction pairs based on task information, the reference result, and the violation type; the incorrect track requires introducing errors relative to the original request and information, and the harmful track requires harmful content or behavior (which may accompany correct task completion).
   • Closeness: prompts only encourage the target to retain the content and structure of the reference result; no distance metric is explicitly optimized and no numerical radius ϵ is enforced; Appendix B gives the full generation prompts.
   • Closeness verification: using Meta-SecAlign as the scoring model, compare the average log-probability gap between the target and the reference result; the authors report that the median gap of the near-target tracks decreases by 52.9–74.2% (incorrect) and 44.7–61.3% (harmful) relative to the fixed-target baseline (Figure 4).

   Target verification

   • incorrect track: for language tasks, an LLM judge checks whether the target introduces errors (prompts in Appendix A); paraphrases and correct alternatives are not retained; for agent tasks, rule-based filtering is used, and targets exactly matching subsequent steps of the ground-truth trajectory are removed.
   • harmful track: an LLM judge evaluates according to a fixed policy Π, where Π is instantiated as harmful standards referencing the official OpenAI and Anthropic usage policies; a correct answer accompanied by leakage or unauthorized actions also counts as a violation.
   • Executability check: agent tool-call targets must use available tools and conform to the parameter schema; Qwen3.8-27B is used as the construction-time executor, executing according to the attack instruction and checking whether the specified target is produced, with failures discarded.
   • Manual verification: LLM checks are combined with manual verification; the two tracks are overlapping evaluation criteria, not a mutually exclusive partition, and success rates are reported separately and not summed.

   Attack execution and success judgment

   • Injection: the instruction is inserted at the specified untrusted content position; for language tasks it is inserted at a random position in the document (all victim models share the same position), and for agent tasks it is inserted at the specified injection slot in the tool return.
   • Judgment: text targets are checked by the judge against the specified deviation across two runs, attacked and unattacked; tool-call targets require the tool called after injection to exactly match with every key parameter (except free-text parameters); the fixed-target baseline uses the benchmark's native checks (PIArena uses a judge, agent suites use the original safety functions).
   • Metrics: deviations already present in the no-injection run are excluded; ASR@3 (the proportion of samples with at least one success in three attempts) and ASR@1 are reported; the attack set is fixed before evaluation, shared across models, and not adapted per model.

   Collapse this answer

4. 04What experiments did the paper conduct?

   On 7 models, the ASR@3 of both near-target tracks is entirely higher than the fixed-target baseline, with a maximum improvement of 35.8 percentage points on language tasks.

   Experimental setup

   • Models tested: Meta-SecAlign-70B (an open-source model trained specifically to resist prompt injection), GPT-5.6 Sol, Gemini 3.7 Flash, Claude Opus 5, GLM-5.3, DeepSeek-V4.1-Flash, Kimi K3, all accessed via official APIs.
   • Datasets: language tasks use 10 datasets from PIArena (Dolly Closed QA/Extraction/Summarization, SQuAD v2, Multi-News, NQ, HotpotQA RAG and long-document, Qasper, LCC), with each sample used in full; agent tasks use 4 suites from AgentDojo v1.2.2 and 3 suites from AgentDyn.
   • Baseline: fixed-target attacks, taking 3 from PIArena's 1,700 original injections or the suites' injection tasks by fixed seed.
   • Judgment: all LLM judgments use GPT-5.6 Luna (with reasoning disabled); tool-call targets require the tool and key parameters to exactly match; the baseline uses the benchmark's native checks.
   • Budget and sampling: attack execution budget B=3 per sample; language tasks have 4,197 attempts in total per track/baseline; temperature is set to 0 (Claude Opus 5 does not support this parameter, Kimi K3 requires 1); agent tasks use the default agent loop with 15 tool-call iterations.
   • Sample unit: task instances for language tasks; reached injection positions for agent tasks (positions never read in the no-attack run are excluded).

   Main results

   According to Table 2 and Table 3 (each cell is ASR@1 / ASR@3, %), Overall row:

   Swipe horizontally to read the table

   Model	Language: fixed	Language: incorrect	Language: harmful	Agent: fixed (Dojo/Dyn)	Agent: incorrect (Dojo/Dyn)	Agent: harmful (Dojo/Dyn)
   Meta-SecAlign-70B	1.1/3.2	17.8/36.7	8.5/18.4	1.6/1.6; 4.2/6.3	9.6/21.5; 15.3/31.6	2.2/5.2; 12.6/21.1
   GPT-5.6 Sol	0.1/0.4	15.9/33.0	6.3/13.9	0.8/2.3; 1.1/2.1	13.7/32.4; 14.4/32.0	2.1/5.7; 9.5/14.7
   Gemini 3.7 Flash	0.1/0.3	3.7/9.6	1.5/4.3	0.0/1.5; 1.0/1.0	5.4/10.9; 8.9/16.8	2.1/4.1; 2.0/4.0
   Claude Opus 5	0.8/2.4	7.6/19.2	1.5/4.1	0.7/1.5; 0.0/0.0	3.4/7.5; 6.2/16.5	1.4/2.8; 2.0/4.1
   GLM-5.3	0.3/0.9	15.2/35.7	5.1/12.7	0.7/4.4; 0.0/0.0	13.1/27.6; 16.0/41.0	5.7/12.8; 10.1/17.2
   DeepSeek-V4.1-Flash	6.0/15.5	27.1/51.3	16.7/32.8	2.2/5.9; 1.0/1.0	16.4/32.2; 17.0/43.4	4.2/8.3; 3.8/10.4
   Kimi K3	0.4/1.1	16.4/36.2	6.2/14.5	1.5/1.5; 0.0/0.0	6.2/16.4; 14.7/35.3	3.5/8.3; 5.9/11.8

   • Language tasks: the authors report that the incorrect track ASR@3 exceeds the baseline on all 70 model-dataset combinations, with improvements of 9.3–35.8 percentage points; the harmful track is higher on 63 combinations, tied on 2, and lower on 5, with improvements of 1.7–17.3 percentage points.
   • Single attempt: the authors report that for every model, the overall incorrect track ASR@1 exceeds the fixed-target ASR@3.
   • Model ranking changes: the authors point out that the two tracks give different model rankings—Gemini 3.7 Flash has the lowest incorrect ASR@3 (9.6%), and Claude Opus 5 has the lowest harmful ASR@3 (4.1%), indicating that a low ASR for one type of target does not imply the same for the other.

   Target validity and judgment reliability

   • Target validity: three experts independently annotated and majority voting served as the manual reference, auditing 150 retained targets per setting-track combination (Table 4); 91.3% of language incorrect and 98.0% of agentic incorrect were confirmed to introduce errors, and 86.7% of language harmful and 92.0% of agentic harmful met the harmful standard.
   • ASR judgment reliability: 100 language task execution records were audited per attack type (10 random records from each of the 10 datasets) (Table 5); the judge's F1 against manual majority voting was 83.6% for fixed targets, 86.7% for incorrect, and 89.9% for harmful, with agreement rates of 91.0%–93.0% and Cohen's κ of 0.78–0.85.

   Other ablations and analyses

   • Target closeness: using Meta-SecAlign as the scoring model, the median average log-prob gap between near-target track targets and reference results decreases by 52.9–74.2% (incorrect) and 44.7–61.3% (harmful) relative to the baseline (Figure 4).
   • Defense-specific model: Meta-SecAlign-70B's ASR@3 rises from a baseline of 3.2% to 36.7% (incorrect) and 18.4% (harmful) (Table 2 Overall row).
   • Single-dataset high point: GPT-5.6 Sol on LCC has a fixed-target ASR@3 of 0.0%, incorrect of 56.6%, and harmful of 49.5% (Table 2).
   • Agent tasks: incorrect track ASR@3 is 7.5%–32.4% on AgentDojo and 16.5%–43.4% on AgentDyn; the harmful track's highest values are 12.8% and 21.1%, respectively (Table 3).

   Collapse this answer

5. 05What are some points that can be further explored?

   The authors state that closeness is only encouraged via prompts and no radius is enforced, and that near-target construction under fixed attack targets is left for future work.

   Limitations and future directions noted by the authors

   • Closeness not quantified: the current construction only encourages the target to be close to the reference result via prompts, and does not enforce a numerical radius ϵ; the authors state that future work could develop task-sensitive result distance metrics and study the effects of ϵ, target coverage, and attack budget (Section 7).
   • Extrapolation from a limited attack set: the authors state that under what conditions resistance to a limited test attack set supports conclusions about untested targets in the neighborhood is left for future work (Section 7).
   • Effect under fixed attack targets unknown: NTBench varies the attacker target for stress testing; whether the near-target construction can improve attack success rate when the attacker has a predetermined target is left by the authors for future work (Section 7).
   • Judgment criteria not fully comparable: in the agent comparison, near-target tool calls use an invocation-based check, while the fixed-target baseline uses an environment-state check; the authors state that the reported ASR characterizes failures under their respective criteria, and that comparisons across target families do not prove an improvement in success against fixed targets (Section 7).

   Experimental coverage

   • Models tested: 7 in total, of which Meta-SecAlign-70B is an open-source defense-specific model and the other 6 are accessed via official APIs (Section 5.1).
   • Benchmarks: language tasks use 10 datasets from PIArena; agent tasks use 4 suites from AgentDojo v1.2.2 and 3 suites from AgentDyn (Section 5.1).
   • Attack budget: 3 attack executions per sample; the attack set is fixed before evaluation, shared across models, and not adapted per model; language tasks have 4,197 attempts per track (Section 5.1).
   • Manual audit scale: 150 per setting-track for target validity (600 in total), and 300 execution records in total for ASR judgment reliability, all annotated by three experts with majority voting (Section 5.3).
   • Unreported items: the paper does not state which model was the generator LLM used for target-instruction generation; it does not report the specific clause list of the harmful standard policy Π (only stating that it references the official OpenAI and Anthropic policies).

   Collapse this answer

6. 06Summarize the main content of the paper

   NTBench re-tests prompt injection robustness with near-target attacks; the ASR@3 of all 7 models is higher than the fixed-target baseline.

   NTBench is a red-teaming benchmark for near-target prompt injection, starting from the point that systems may pass existing evaluations by refusing attacks that obviously abandon the task, yet still accept attacks that retain a legitimate appearance but introduce erroneous or harmful changes.

   • Method: anchored on legitimate reference results, use an LLM to generate target-instruction pairs that retain most of the content but introduce errors (incorrect track) or harmful results (harmful track); after LLM checks and manual verification confirm the violation, inject into untrusted content in an addition-only manner; judge success by target-specific checks, excluding deviations already present without injection.
   • Main results: on 7 LLMs, 10 PIArena language datasets, and AgentDojo and AgentDyn, the ASR@3 of both near-target tracks is higher than the fixed-target baseline for every model; language task improvements are 9.3–35.8 percentage points (incorrect) and 1.7–17.3 percentage points (harmful) (Table 2, Table 3).
   • Defense-specific model: Meta-SecAlign-70B, trained specifically to resist prompt injection, rises from a baseline ASR@3 of 3.2% to 36.7% (incorrect) and 18.4% (harmful).
   • Ranking changes: the authors point out that the two tracks give different model rankings, and a low ASR for one type of target does not imply the same for the other (e.g., Gemini 3.7 Flash has the lowest incorrect ASR@3, and Claude Opus 5 has the lowest harmful ASR@3).
   • Reliability: manual auditing confirms that 86.7%–98.0% of retained targets indeed constitute erroneous/harmful deviations; the agreement rate between the judge and manual majority voting is 91.0%–93.0%.
   • Authors' conclusion: resistance to fixed-target attacks does not imply resistance to near-target erroneous/harmful results, and reference-guided target construction can expand the failure surface revealed by robustness evaluations; quantifying closeness and near-target construction under fixed attack targets are left for future work.

   Collapse this answer

Related papers