Invisible to humans, triggered by agents: a stealthy jailbreak attack against mobile vision-language agents

Core question: how to carry out jailbreak prompt injection against LVLM-driven mobile agents that goes unnoticed by human users, under real mobile deployment conditions with no system-level privileges and only one injection opportunity.

• Scenario and importance: LVLM mobile agents operate sensitive data on the device and can autonomously initiate real actions; the authors believe that once compromised, this could lead to privacy leakage, financial loss and safety risks.
• Shortcomings of existing methods: the authors say visual manipulation such as transparent overlays is conspicuous in real environments and easily noticed by users; poisoned-icon attacks assume adversarial control over UI resources; OS-level injection channels require elevated privileges and have limited robustness against LVLM built-in safety alignment; pixel-level imperceptible perturbations transfer poorly to unseen models (especially commercial closed-source LVLMs).
• Core observation: the touch signals (size, pressure) produced by automated agents are near zero and consistently differ from human touch patterns, which can be used to distinguish the source of interaction; moreover, mobile agents typically decide based on a single screenshot, so the attacker has only one chance and is limited by on-screen text length.
• The paper proposes: a new attack paradigm, agent-only perceptual injection (malicious content is exposed only during agent interaction), and a unified framework (in-app embedding, interaction-triggered activation, and the HG-IDA* one-shot jailbreak optimization method).
• Threat model (Section 3.1):
  • Goal: induce the mobile agent to execute attacker-specified instructions rather than the user's intent.
  • Knowledge: the paper does not clearly state the attacker's level of knowledge about the target LVLM; detoxification uses open-source LlamaGuard3 as the safety scoring model, and the target model's content moderation system is described as closed-source and opaque.
  • Capability: no system-level privileges (no overlay or accessibility), can distribute malicious/repackaged apps via third-party markets or sideloading, and controls content rendered within the app.
  • Victim system: a mobile agent running under restricted permissions, interacting with the app through screenshots and standard touch events (e.g., ADB-generated input in a controlled environment).

Mobile agents

• AppAgent (Zhang et al., 2025b) — a multimodal framework that controls Android apps by having an LLM generate action plans based on GUI screenshots.
• Mobile-Agent (Wang et al., 2024b) and Mobile-Agent-V (Wang et al., 2025a) — improve robustness through action correction and multi-agent collaboration.
• Mobile-Agent-E (Wang et al., 2025b) — a multi-sub-agent architecture separating perception, planning and execution, aimed at long-horizon complex tasks; the experiments in this paper are based on this framework. InfiGUIAgent, ClickAgent, Mobile-Agent-V2 and others adopt a similar vision-language model plus system-level API architecture.

Security of multimodal mobile agents

• WIPI (Wu et al., 2024b), EIA (Liao et al., 2024) — prompt injection attacks against web/desktop agents, embedding adversarial text into pages or dialog boxes.
• Wu et al. (2025) — systematically analyze the attack surface of mobile agents, demonstrating GUI hijacking via transparent overlays, pop-ups, etc.; the paper notes that these attacks rely on conspicuous UI changes, require overlay permissions and lack covert trigger strategies.
• Yang et al. (2024) — propose a security matrix for mobile multimodal agents, demonstrating adversarial UI elements such as poisoned icons; the paper notes that their threat model assumes full control over UI assets.
• AEIA (Chen et al., 2025) — injects malicious prompts via system notifications; the paper notes that it relies on privileged access to the notification channel and does not demonstrate success in bypassing LLM safety filters. Aichberger et al. (2025) show that image perturbations imperceptible to humans can hijack agent behavior, but the authors say such perturbations transfer poorly to unseen models (especially commercial closed-source LVLMs).

Jailbreak attacks

• Single-shot non-iterative techniques: the DAN family (Shen et al., 2024) uses role-playing to induce models to ignore safety guardrails; the white-box optimization method GCG (Zou et al., 2023) uses gradients to generate adversarial suffixes that can be generated offline and transferred for one-time use.
• Automated jailbreak generators: AutoDAN (Liu et al., 2023) and GPTFuzz (Yu et al., 2023) rely on multi-step search, large query budgets or stronger access (white-box gradients or external LLM evaluators), which the authors consider incompatible with the one-shot threat model of this paper.

Baselines and benchmarks: the baselines compared in the experiments are Direct Ask (DA), the Prefix attack (Shen et al., 2024), and the GCG constant suffix (Zou et al., 2023, taken from the HarmBench codebase and optimized on a small model); evaluation uses the authors' own dataset of 40 injection instances.

Positioning: the authors say that, to their knowledge, no existing work has studied low-privilege, covert and one-shot jailbreaks of mobile agents under realistic UI constraints, and that this paper's framework covers all three points: low-privilege operation, covertness and single-shot effectiveness.

Overall idea: pre-embed the jailbreak prompt δ into the interface elements of a malicious app, exploit the difference between human and agent touch signals to render it into the screenshot only during agent interaction, and use HG-IDA* to optimize the prompt under single-round and length constraints to bypass safety filters. The framework is called agent-only perceptual injection.

Problem formulation

• Let st be the agent's visual observation and et an interaction event; when et−1 satisfies the activation condition, the app renders the payload δ, producing a modified observation s′_t = Act(s_t, δ, e_t−1), where Act(·) is the in-app activation operator.
• The attack goal is to find δ^⋆ = argmax_δ Pr[π(s′_t^⋆) ∈ A_Ga ∣ e_t^⋆−1 = 1], i.e., under agent-driven interaction, maximize the probability that the agent policy's output falls into the attacker's target action set; D is the feasible in-app text payload space.

In-app embedding and interaction-triggered activation

• Unprivileged perceptual compromise: δ is embedded directly into standard interface elements within the app's rendering hierarchy, without relying on overlays, accessibility services or system-level rendering hooks, so no elevated privileges are needed.
• Agent-attributable activation: take the raw touch record rt = (sizet, pressuret) and use a simple classifier: when sizet ≤ ϵs or pressuret ≤ ϵp, it is judged as agent interaction (et = 1), otherwise human interaction (et = 0); ϵs and ϵp are small thresholds characterizing the near-zero-contact features of ADB-generated touches.
• When et−1 = 0 the observation is unchanged; when et−1 = 1, δ is rendered for exactly one step, aligning the exposure window with the agent's control flow. Appendix B gives the detection pseudocode (Algorithm 1, thresholds size < 0.01 or pressure < 0.05).

HG-IDA*: one-shot jailbreak optimization

• Two-stage process: (1) templating, assembling atomic components into action-oriented templates; (2) character-level detoxification, applying minimal perturbations to tokens to raise the safety score S(·) while preserving semantic similarity to the original intent.
• Four atomic components: Hook (a short attention-guiding fragment), Instr (the original instruction or its detoxified variant), Jail (a sub-template selected by direct-execution or content-generation type), Distract (a short benign phrase masking the malicious intent). When the instruction appears harmless, only Hook and δ0 are concatenated; when clearly malicious, Hook, δ, Jailtype(δ) and Distract are concatenated. Appendix D gives the full templates.
• Detoxification search: for the top-N words with the highest harmful attribution in the safety classifier score, apply at most one character-level edit each; the safety probability S(·) is computed from LlamaGuard3 logits, Sim(·,·) is cosine similarity; the heuristic gain is h(s) = wsafety·∆P(s) + wsim·∆Sim(s), with constraints S(s) ≥ τ, Sim(s, δ0) ≥ γ, |T(s)| ≤ Lmax.
• IDA* and pruning: iterative deepening over the sentence-level edit budget g ∈ {0, ..., Dmax}, expanding each layer in descending order of h(·); each layer maintains a bounded min-heap of size at most Kchain and sets a warmup window W; candidate nodes are committed only after downstream search validation, and uncommitted ones are rolled back. Full pseudocode is in Appendix C (Algorithm 2).

Key parameters and success criteria

• Parameters (Appendix A): wsafety = 0.9, wsim = 0.1; Kchain = 5; W = 20; Dmax = 3; γ = τ = 0.8; at most one character edit per word (substitution, insertion, deletion), sampling at most V = 7 candidates per position.
• Success criteria: Trigger Detection Accuracy measures the proportion of ADB touches correctly identified as automated; Tasr (Thought ASR) measures whether the injected prompt enters the agent's internal planning, and Rasr (Result ASR) measures whether the malicious plan is actually executed in the environment; both are evaluated manually.

Experimental setup

• Agent framework: Mobile-Agent-E (a modular multi-agent architecture separating perception, planning, and execution), which interacts via ADB touch events and takes screenshots at each decision point.
• Backends tested (6): GPT-4o-2024-11-20, Gemini-2.0-pro-exp-0205, Claude-3-5-sonnet, Qwen-vl-max, Deepseek-VL2, Llava-OneVision-Qwen2-72b-ov-Chat.
• Dataset: 40 self-built prompt-injection instances (including both obviously malicious and seemingly benign categories), distributed across three Android apps, WeChat, Memo, and SmartHome; the authors plan to release the dataset in a desensitized, controlled manner.
• Baselines: Direct Ask (DA), Prefix (Shen et al., 2024), GCG constant suffix (Zou et al., 2023, optimized on a small model provided in the HarmBench codebase).
• Metrics and review: Tasr, Rasr, and Trigger Detection Accuracy, defined in Section 4.1; attack results are manually evaluated.
• Sample size and repetitions: 40 instances per backend; the paper does not report the number of repetitions.

Main results

Table 1: Tasr / Rasr (%) for each backend on 40 instances (split into Harmful / Harmless / Total).

Swipe horizontally to read the table

Model	Tasr Harmful	Tasr Harmless	Tasr Total	Rasr Harmful	Rasr Harmless	Rasr Total
GPT-4o	75.0	93.8	82.5	66.7	87.5	75.0
Gemini-2.0-pro-exp-0205	95.8	93.8	95.0	91.7	68.8	82.5
Claude-3-5-sonnet	8.3	81.3	37.5	4.2	56.3	25.0
Qwen-vl-max	91.7	100	95.0	4.2	6.3	5.0
Deepseek-VL2	79.2	100	87.5	20.8	31.3	25.0
LLaVA-OneVision	37.5	75.0	52.5	33.3	50.0	40.0

• The authors state that mobile agents are vulnerable to single-shot perception-chain prompt injection attacks under real device conditions, and that current LVLM mobile agents lack security guarantees in real-world scenarios.
• The authors observe two patterns: some highly capable closed-source models have both high Tasr and high Rasr (GPT-4o, Gemini-2.0); some open-source or less capable backends have high Tasr but low Rasr (Qwen-vl-max, Deepseek-VL2).
• The authors attribute the Tasr–Rasr gap to backend heterogeneity: highly integrated models can reliably translate plans into actions, while weaker or less integrated models fail at grounding or tool invocation.

Jailbreak baseline comparison

• Setup: Table 2 compares DA, Prefix, GCG, and HG-IDA* on GPT-4o, Gemini-2.0-pro, Deepseek-VL2, and LLaVA-OneVision across three harmful instruction subcategories: Execute / Generate / Persuade.
• Results: On the Total row, HG-IDA* reaches 75.0% Tasr / 66.7% Rasr on GPT-4o, 95.8% / 91.7% on Gemini-2.0-pro, 79.2% / 20.8% on Deepseek-VL2, and 37.5% / 33.3% on LLaVA-OneVision; the three baselines are all 0.0% on GPT-4o, and at most 58.3% / 25.0% (DA) on Gemini-2.0-pro.
• Authors' interpretation: verbatim queries, role-play prefixes, and GCG suffixes do not reliably transfer to LVLMs with content moderation, whereas this paper's pipeline translates planning-level compromise into a higher end-to-end execution rate.

Ablation experiments

• Setup: Table 3 compares four configurations, DA, w/o template, w/o opt, and Ensemble (full pipeline), on GPT-4o and Deepseek-VL2.
• Results: On GPT-4o, DA is 0.0% / 0.0%, w/o template is 33.3% / 25.9%, w/o opt is 16.7% / 12.5%, and Ensemble is 75.0% / 66.7%; on Deepseek-VL2, DA is 0.0% / 0.0%, w/o template is 4.2% / 4.2%, w/o opt is 8.3% / 8.3%, and Ensemble is 79.2% / 20.8%.
• Authors' interpretation: both the structured framework (template) and targeted obfuscation (detoxification) are necessary conditions for jailbreak success.

Trigger detection and defense evaluation

• Trigger detection (Appendix B, Table 4): the specificity-aware detector's Trigger Detection Accuracy for ADB_Tap is 100 on all three apps, WeChat, SmartHome, and Memo, and 0 for Hand_Tap on all; the results are aggregated from all evaluation experiments, and the paper does not report the per-cell sample size.
• Defense (Appendix F, Table 5): the authors propose a provenance-aware prompting defense (appending authorization markers to user instructions and instructing the agent to execute only commands from authorized sources), evaluated on the GPT-4o backend with 40 instances. User-Defense reduces the Tasr total from 82.5% to 10.0% and the Rasr total from 75.0% to 7.5%; Agent-Defense has Tasr / Rasr totals of 15.0% / 7.5%. The authors state that this low-cost intervention improves instruction attribution.

Other ablations and analyses

• The Appendix F main text states that the defense reduces the previous attack success rates of 75% and 62.5% to 7.1% and 12.5%, and labels them as from Table 4; in the converted text, Table 4 is the trigger detection results and Table 5 is the defense results, and the two sets of numbers cannot be matched, so they are listed here as questionable.
• Section 4.4 Findings: the authors state that the modular architecture expands the attack surface (injected text can be captured by the perception chain and stored in auxiliary modules such as memory); the agent often misattributes injected UI text to the latest user instruction; cross-app navigation amplifies the real-world impact of injection.
• Figure 2 compares Tasr and Rasr across backends and provides textual descriptions of successful/failed jailbreak examples; the figure annotates models with a large Tasr–Rasr gap and models vulnerable at both stages.

Limitations and future directions noted by the authors

• Controlled environment (Section 6): the study focuses on prompt injection attacks embedded in Android apps, evaluated in a controlled mobile agent environment.
• Robustness not fully explored (Section 6): the robustness of the activation and detoxification mechanisms under more device configurations, other agent architectures, or future OS-level UI changes has not been fully explored.
• Distribution assumption (Section 6): the attack assumes the adversary can modify and redistribute the app; the authors state this is realistic for third-party ecosystems but does not apply to all distribution channels.
• Payload length (Section 6): the detoxification search is performed on short, screen-constrained prompts, and significantly longer or multimodal payloads may require additional adaptation.

Experimental coverage

• The models tested are 6 LVLM backends: GPT-4o-2024-11-20, Gemini-2.0-pro-exp-0205, Claude-3-5-sonnet, Qwen-vl-max, Deepseek-VL2, Llava-OneVision-Qwen2-72b-ov-Chat (Section 4.1).
• The agent framework is only Mobile-Agent-E; the apps are three self-built Android apps, WeChat, Memo, and SmartHome; the dataset is 40 injection instances (Section 4.1).
• Baseline comparisons cover three, DA, Prefix, and GCG (Section 4.3); ablations are conducted on two backends, GPT-4o and Deepseek-VL2 (Table 3).
• Defense evaluation covers two variants of provenance-aware prompting, user-side and agent-side, reported on the GPT-4o backend with 40 instances (Appendix F, Table 5); the paper does not report adaptive attacks against this defense.
• Attack success is determined by manual evaluation (Section 4.1); the paper does not report the number of repetitions per backend instance, the per-cell sample size for trigger detection, or the consistency of the manual evaluation.

An attack paper targeting mobile LVLM agents that proposes the agent-only perceptual injection paradigm: exploiting the difference that automated agent touch signals are near-zero and differ from human touch patterns, it pre-embeds malicious prompts into the interface of a malicious app, rendering them only during the brief window of agent interaction while remaining invisible to human users; the attack requires no system-level privileges and fits the one-shot constraint of single-screenshot decision-making.

• Problem to solve: existing mobile agent injection attacks either rely on conspicuous persistent visual manipulation or require privileged access such as overlays and notification channels, while pixel-level perturbations are hard to transfer to closed-source LVLMs; low-privilege, stealthy, one-shot jailbreaks remain open in real mobile scenarios.
• Method highlights: a three-component framework—in-app embedding of standard UI elements, interaction-triggered activation based on touch size/pressure thresholds (exposure of exactly one step), and HG-IDA* optimization (a four-component template of Hook/Instr/Jail/Distract plus a character-level detoxification search scored by LlamaGuard3, with IDA* iterative deepening and top-K pruning).
• Main results: on Mobile-Agent-E with 40 injection instances, the GPT-4o backend reaches 82.5% Tasr / 75.0% Rasr, and Gemini-2.0-pro-exp-0205 reaches 95.0% / 82.5% (Table 1); Qwen-vl-max shows a gap of 95.0% Tasr versus 5.0% Rasr (Table 1). HG-IDA* exceeds the DA, Prefix, and GCG baselines on all four backends (Table 2). Trigger detection achieves 100 accuracy for ADB_Tap (Table 4).
• Defense: the authors propose provenance-aware prompting, which on GPT-4o reduces the total Rasr value from 75.0% to 7.5% (user-side and agent-side identical, Table 5).
• Authors' conclusion: low-privilege attackers can manipulate trusted mobile agents through stealthy in-app visual injection, and highly capable closed-source models are more likely to turn hijacked plans into actual harmful actions; the authors argue defenses need to incorporate interaction-level signals, and note that the robustness of the activation and detoxification mechanisms under more device configurations, other agent architectures, and future OS-level UI changes has not been sufficiently explored.