MobiRed framework evaluates prompt injection attacks via third-party channels on eight mobile LLM agents

This paper attempts to address the security risk of mobile LLM agents being indirectly manipulated as confused deputies when facing third-party interaction channels not controlled by the app.

• Mobile Agent Application Scenarios and Security Sensitivity: Mobile LLM agents autonomously navigate mobile apps and orchestrate cross-app workflows through a "perceive-think-act" loop. Because agents have broad user-level permissions to observe screens, operate UIs, and invoke system APIs, if their behavior is manipulated, it directly endangers sensitive device data.
• Inadequacy of Existing Security Evaluation Settings: Existing agent security benchmarks mainly focus on web or desktop environments; a few existing security studies on mobile agents assume the attacker controls a malicious app on the device, but in reality, an attacker who controls a high-privilege app lacks the motivation to attack the agent.
• Core Observation and Hypothesis: Mobile agents must process trusted user instructions and unverified dynamic environmental content (in-app ads, embedded web pages, notifications, or SMS) in a single model context. An attacker does not need to control the device; by merely injecting adversarial content through legitimate third-party content channels, they can induce the agent to misclassify malicious content as task instructions, becoming a confused deputy that abuses its own privileges.
• Threat Model:
  • Attacker Goals: Manipulate the agent to deviate from the user's benign goals and achieve attacker-specified objectives, including leaking credentials/OTPs/personal privacy, performing unauthorized cross-app actions (clicking ads, submitting forms), and downloading and installing untrusted software.
  • Attacker Knowledge and Permissions: The attacker is assumed not to control the victim device, mobile OS, firmware, apps, or agent implementation; has no root access; cannot modify model weights, system prompts, or execution logic; and does not directly insert malicious text into the user's original input.
  • Attacker Capabilities: The attacker has legitimate access to common third-party channels in the ecosystem (in-app ads, embedded WebViews, external web pages, email, SMS, cross-app notifications), can control the content, wording, formatting, and timing of injected payloads, and can host attacker-controlled endpoints or landing pages.
  • Victim System: An uncompromised Android device with a benign user issuing normal task instructions, running a mobile LLM agent with standard system permissions.
• Solution Proposed in This Paper: The paper designs an automated attack and testing framework, MobiRed, to systematically evaluate the vulnerability of 8 mainstream mobile LLM agents under non-app-controlled channels.

Mobile LLM Agent Architectures and Automation

• Perception and Control Paradigms: AutoDroid (Wen et al., 2024) automates single-app tasks based on accessibility trees and program synthesis; AppAgent (Zhang et al., 2025) uses a vision-driven dual-module (OmniParser and Planner) to execute screen taps; Mobile-Agent-E (Wang et al., 2025) and AutoGLM (Liu et al., 2024) adopt hierarchical multi-agent architectures to support cross-app task orchestration.
• Benchmark Evaluation Environments: AndroidWorld (Rawles et al., 2025) proposes M3A and T3A agents and provides a dynamic mobile evaluation environment; Mobile-Bench (Deng et al., 2024) and SPA-Bench (Chen et al., 2024) focus on evaluating mobile agents' task completion rates and usability. The authors note that these benchmarks primarily focus on task completion capability, not adversarial robustness.

Untrusted Third-Party Content in the Mobile Ecosystem

• Third-Party SDK and Ad Abuse: Zhan et al. (2021) systematically investigate security issues in Android third-party libraries; Maddroid (Liu et al., 2020) and Sun et al. (2021) analyze malvertising and click fraud; Son et al. (2016) study mobile ad tracking and permission boundaries. The authors note that traditional attacks target human users or vulnerable apps, whereas in the LLM era, third-party content directly becomes environmental input to agents.
• Cross-Channel Social Engineering and WebView Risks: Nahapetyan et al. (2024) and Agarwal et al. (2025) study smishing infrastructure and fraud; Babelview (Rizzo et al., 2018) and Chin & Wagner (2013) analyze WebView and JavaScript bridge vulnerabilities.

LLM Agent Security and Prompt Injection Benchmarks

• Environment Interaction and Tool Call Security: ToolEmu (Ruan et al., 2023) and R-Judge (Yuan et al., 2024) evaluate agent risks in controlled sandboxes; AgentDojo (Debenedetti et al., 2024) and Agent Security Bench (Zhang et al., 2024) propose dynamic indirect prompt injection evaluations for tool and browser agents.
• Mobile Agent Security Research: Wu et al. (2025) explore security risks of mobile LLM agents, but the authors note that this work assumes the attacker already controls an installed app on the victim's device.

Baseline Methods and Evaluation Benchmarks

• Agents Under Test and Benchmark Setup: The experiments select 8 mainstream mobile agents as evaluation targets, including AutoDroid, Droidrun, AutoGLM, AppAgent, M3A, T3A, SeeAct, and Mobile-Agent-E; tasks are built on real apps such as Fossify Notes, Calendar, File Manager, Recorder, and Clock; injection attack patterns are adapted from works such as InjecAgent, AgentDojo, and Agent Security Bench.

Positioning Differences Between This Paper and Existing Work The authors emphasize that existing agent security benchmarks mainly target web or desktop tool calls and do not consider mobile-specific heterogeneous third-party channels and interactions with OS security defenses; whereas existing mobile security research assumes overly strong attacker control. This paper, under a strict and practical threat model where the attacker controls no on-device resources, systematically evaluates the vulnerability of mobile agents as confused deputies.

The authors design and implement an end-to-end automated security evaluation framework for Android mobile LLM agents, MobiRed. The framework compiles high-level attack specifications into executable adversarial test cases, executes them in real mobile environments, injects third-party adversarial content, and achieves automated evaluation and attribution through three-stage behavior metrics.

Problem Setting and Formalization

Mobile LLM agents execute tasks through iterative interaction: the trusted instruction context is denoted as p (including system prompts and user task instructions). At each interaction step, the agent collects environmental observations dt (e.g., screenshots, accessibility trees, or UI view hierarchies). The backend foundation model F receives the concatenated context and generates device action decisions, formally expressed as: a_t = F(p ∥ d_t) In this decision formula, at represents the next device operation generated by the model based on the concatenated context. When unsanitized third-party content is mixed into the environmental observation dt, the foundation model cannot effectively distinguish the trust boundary between p and dt, and malicious content injected by the attacker is mistakenly executed as task-related instructions, causing the agent to become a confused deputy.

Adversarial Scenario Generator

The scenario generator compiles structured YAML configuration files into executable adversarial test cases, each pairing a benign user task with a controlled injection event. The YAML specification includes four core modules:

• Test Context: Defines the baseline execution environment, including the agent under test, target app, and benign user goal (e.g., instructing the agent to create a note with specific content in a notes app).
• Attack Workflow: Combines MITRE ATT&CK for Mobile, OWASP Top 10 for Agentic Applications, and NIST AI RMF, covering 14 mobile adversarial tactics and core agent risks, with predefined complete action sequences from initial trigger to privilege escalation or data exfiltration.
• Attack Pattern: Parametrically integrates 10 prompt injection patterns (P1–P10), including Direct Harm, TODO attack, Naive Attack, Escape Characters, Fake Completion, Context Ignoring, Combined Attack, InjecAgent Benchmark, Important Message, and App Specific.
• Attack Distribution Channel: Covers two types of third-party channels: one is external social engineering channels (including email, SMS, and WhatsApp message pushes) used to induce cross-app jumps; the other is in-app integrated ad channels (integrating banner ads, splash ads, and interstitial ads via the Google Mobile Ads SDK).

Instrumented Android Testbed

The testbed is responsible for executing generated adversarial test cases under controlled and realistic mobile conditions:

• Environment Management and Standardization: Automatically restores the Android device to a clean initial state before each test run and deploys a controlled host app integrated with the ad SDK in specified app scenarios.
• Multimodal Telemetry Collection: The testbed synchronously records three types of telemetry data to support automated judgment:
  1. Agent logs: records model inputs (screenshots, accessibility trees, OCR text), intermediate chain-of-thought planning, and timestamped UI operations (clicks, text inputs, cross-app switches);
  2. Attacker backend telemetry: deploys a controlled HTTP backend that receives and records ad click redirects, fake login form submissions, and sensitive data exfiltration via a trial_id uniquely bound to the test case;
  3. System event monitoring: monitors underlying Android system events, including APK installation and uninstallation status changes, system permission grant dialog confirmations, system setting changes, and file write records.

Automated Attack Evaluator

The evaluator receives the testbed's multidimensional telemetry data packets and decomposes each test case's execution status into three-stage behavior indices (Agent Behavior Index):

• Notice Stage (N): Detects whether the malicious payload enters the agent's perception input. By extracting the model input context and invoking an LLM-based detector, it determines whether the injected content is visible (output YES/NO).
• Attempt Stage (A): Detects whether the agent produces decision intentions that advance the attack goal. An LLM-based classifier analyzes the agent's reasoning chain and high-level plan to determine whether its goal aligns with the attack intent.
• Execute Stage (E): Based on dual real evidence from on-device system events and attacker backend logs, determines whether verified actual security harm has occurred. E=1 is assigned only when all required steps (e.g., clicks, redirects, form submissions, or system installations) are completed; if stalled or failed midway, E=0 is assigned. Validated on 800 manually annotated test logs, the perception detector achieves an F1 of 0.9535, the intent classifier achieves an F1 of 0.9424, and the event monitor achieves an F1 of 0.9987.

Experimental Setup

• Tested Agents and Backbone Models: Evaluated 8 mainstream mobile LLM agents, including Mobile-Agent-E, AppAgent, AutoGLM, Droidrun, M3A, T3A, SeeAct, and AutoDroid. The backbone models used cover GPT-3.5-turbo to GPT-5, Claude-3.5-Sonnet, Gemini-1.5-Pro / 2.5-Flash, Llama, and Qwen series (Table 5, Table 11).
• Test Environment and Benchmark Applications: Ran on Pixel 3–9 series emulators covering Android 9–16 (API 28–36) and on Pixel 6a and HONOR ANN-AN00 physical devices running Android 15; involved 5 daily open-source applications (Fossify Notes, Calendar Planner, Simple File Manager Pro, Simple Voice Recorder, Simple Clock) as well as system and third-party applications (Chrome, Gmail, WhatsApp, Settings, Messages, Google Play).
• Attack Workflows and Vectors: Designed 8 types of attack workflows, covering low-friction vectors (V1 fraudulent ads, V2 phishing content leakage, V3 fake login phishing), OS protection vectors (V4 malware deployment, V5 malicious app deployment), and long-chain cross-app vectors (V6 clipboard/OTP theft, V7 cross-app data hopping, V8 system data discovery) (Table 1, Table 8).
• Evaluation Metrics: Benign task success count (S), Notice count (Notice, N), Attempt count (Attempt, A), and final execution success count (Execute, E), all presented as success counts out of 10 repeated tests (e.g., 7/10), with aggregated attack success rate (ASR) also reported.
• Review and Judgment Method: Notice (N) is determined by extracting multimodal input and judging with an LLM detector; Attempt (A) is determined by an LLM classifier analyzing the agent's reasoning plan; Execution (E) is hard-determined based on attacker backend network reception logs and Android system event logs (installations, permission grants, file operations).
• Test Scale and Repetition: Each test condition was repeated 10 times to eliminate uncertainty; the full experiment accumulated 3,160 rounds of adversarial testing and paired benign testing, taking about 400 device hours and consuming about $240 in API costs.

Main Results

Swipe horizontally to read the table

According to Table 2 (showing each model's execution success count E/10 out of 10 tests; attack vectors omit V2 content leakage, V6 OTP theft, and V7 cross-app data hopping).

• Low-friction attacks show broad effectiveness: The authors state that V1–V3 vectors require only 2–4 steps and no bypass of system protections, with average notice, attempt, and execution across agents reaching 9.38, 7.00, and 5.54; V1 fraudulent ads average 8.63/10 executions, and V3 fake login averages 51.25% execution rate.
• System-level security protections cannot block high-capability agents: The authors state that V4 and V5 require bypassing OS-level warnings such as harmful download and permission confirmations, but Mobile-Agent-E (V4 execution 9/10) and AppAgent (V4 execution 6/10), which have end-to-end planning capabilities, can autonomously close pop-ups and complete installation; M3A and SeeAct, although generating attack attempts (A=10), stall at the final download button or permission pop-up due to UI interference.
• Long-chain cross-app attacks are currently limited by agent execution capability: The authors state that V6–V8 involve cross-app state retention and context switching, with only Mobile-Agent-E able to execute reliably (V8 reaching 9/10), while single-app architectures AutoDroid and AppAgent fail completely; M3A, although noticing multiple times (N=10) and attempting (A=7), fails to complete end-to-end attacks due to difficulty in long-sequence execution.

Distribution Channel Comparison: In-App Channels vs. Cross-App Channels

• Evaluation Setup: Under the V1 fraudulent ad vector, compared four in-app channels—banner ads (E1), splash ads (E2), interstitial ads (E3), WebView (E4)—and two cross-app channels—email (E5) and WhatsApp message notifications (E6)—across 8 agents (Table 3).
• Experimental Results: In-app channels show higher average attack performance than cross-app channels, with average notice, attempt, and execution of 9.88, 6.46, and 6.33; among them, interstitial ads have the highest execution rate (average E reaching 86.25%), and splash ads have the lowest execution rate (average E of 17.50%); among cross-app channels, email and WhatsApp average execution is only 12.50% and 11.25%, but on Mobile-Agent-E they still reach 7/10 and 9/10 executions.
• Author Interpretation: The authors state that interstitial ads pop up mid-execution and occupy the center of the screen, effectively interrupting the agent's original flow; splash ads appear before task start, making it easier for the agent to close them; cross-app notifications, although easily noticed, are often judged by the agent as irrelevant information or limited by cross-app execution capability.

Attack Prompt Patterns and UI Complexity Effects

• Evaluation Setup: Fixed the notes task and banner ad channel, compared 10 injection patterns (P1–P10) and benign ad P0 (Table 9, Figure 3); tested the transferability of combined attacks across 5 application scenarios with different UI complexity (notes, calendar, file manager, voice recorder, clock) (Table 10, Figure 4).
• Experimental Results: Under benign ads, 7/8 agents were not diverted; among malicious patterns, short and direct instruction attacks have the highest success rate (Naive Attack P3 about 60%, Fake Completion P5 about 55%), while structurally complex InjecAgent (P8) and Important Message (P9) are below 27%; across the 5 application scenarios, the average cross-scenario attack success rate is 39.5%, with low-complexity interfaces (notes, voice recorder, 10–20 UI elements) averaging about 53.75% attack success, while high-complexity interfaces (calendar, file manager) show decreased attack success.
• Author Interpretation: The authors state that agents tend to obey short, clear inputs, and lengthy instructions introduce goal conflicts; dense UI elements increase the difficulty of noticing and locating, causing both benign tasks and malicious instructions to fail easily.

Backbone Model Capability-Risk Coupling Phenomenon

• Evaluation Setup: Under the M3A framework and notes task, keeping the task and V1 combined attack unchanged, only swapped 12 commercial and open-source backbone LLMs (GPT-4.1 variants, GPT-4 Turbo, GPT-5, DeepSeek-v3/r1, Llama-4-Maverick, Gemini-2.5-Flash variants, Qwen series) (Figure 5).
• Experimental Results: Most high-performance models cluster in the upper-right region, with high benign task success accompanied by high attack success; two exceptions are GPT-5, which maintains high benign completion while having lower attack success, and Qwen2.5-VL-7B, where both benign task and attack success are low.
• Author Interpretation: The authors state this exhibits capability-risk coupling: as backbone models' ability to follow complex instructions increases, their ability to execute malicious injected instructions also increases in tandem; GPT-5 shows a stronger tendency toward ad recognition and deprioritization, while small models' low attack success stems from their own capability limitations rather than robust defense.

Other Ablations and Analyses

• Validation of automated evaluators on 800 human-annotated logs: injection detector Precision 0.9516, Recall 0.9554, F1 0.9535; intent classifier Precision 0.9171, Recall 0.9691, F1 0.9424; event monitor Precision 0.9974, Recall 1.0000, F1 0.9987 (Table 6).
• Association between attack sequence complexity and execution rate: 2–4 step low-friction attacks average 5.54/10 executions; 6–7 step attacks requiring system protection bypass average 2.56/10 executions; 6–8 step cross-app attacks with 2–3 context switches average 1.04/10 executions (Table 2, Table 8).
• AutoGLM's defensive performance in complex adversarial flows: completed 88 of 90 benign test rounds (97.8%), but achieved 0/10 executions in V4 malware, V6 OTP theft, and V7 cross-app hopping (Table 2).
• Droidrun's robust defensive behavior: with high benign cross-app task success, execution success is 0/10 for the other 7 attack vectors except V1 fraudulent ads (execution 4/10) (Table 2).
• Attribution of failure cases and anomalous behavior: AutoDroid fully resisted V1 (E=0) but reached 10/10 executions in V2; the authors explain that the forged notes interface closely matches the benign flow; SeeAct had an attempt rate of 10/10 in V4 but 0/10 executions; the authors explain that ad interference caused it not to click the correct download button; AppAgent attempted 9 times in V3 but had 0 executions; the authors explain that a mismatch between model reasoning and action space led to invalid commands.

Limitations and Future Directions Noted by the Authors

The paper does not specifically discuss limitations.

Experimental Coverage

• Scope of Tested Agents and Backbone Models: The experiments tested 8 mobile agent architectures: AutoDroid, Droidrun, AutoGLM, AppAgent, M3A, T3A, SeeAct, and Mobile-Agent-E; in the backbone model analysis, 12 models were tested: GPT-3.5-turbo through GPT-5, DeepSeek-v3/r1, Gemini-2.5-Flash series, Llama-4-Maverick, and Qwen series (§VI, Table 5, Figure 5).
• Device Environments and OS Versions: The test environments included Pixel 3–9 series emulators covering Android 9 to Android 16, as well as physical devices Pixel 6a and HONOR ANN-AN00 running Android 15 (§VI, Table 5).
• Attack Vectors and Interaction Channels: The attack workflow covers 8 scenarios (V1 fraudulent ads, V2 content-leak phishing, V3 fake login phishing, V4 malware deployment, V5 malicious app deployment, V6 clipboard/OTP theft, V7 cross-app data exfiltration, V8 system data discovery); channels include in-app (banner, splash, interstitial, WebView) and cross-app (email, WhatsApp message notifications) (§IV, §V-A, Table 1, Table 3).
• Sample Size and Repetitions: Each test condition was repeated 10 times, with a total of 3,160 adversarial test runs across all experiments; the evaluator validation included 800 manually annotated logs (§VI, Table 6).
• Evaluation and Defense Design: The study tested the interception effect of existing mobile OS defenses (harmful file download warnings, installation confirmations, and permission dialogs) on agent execution; the paper did not evaluate dedicated prompt injection defense algorithms or guardrail mechanisms (§VI-A).

• Positioning and core problem: This study evaluates the security of mobile LLM agents in the Android environment against third-party non-app-controlled channels. Agents must accept unverified environmental content while executing tasks; without strict trust boundaries, attackers can induce agents to abuse their own high privileges to execute malicious actions without controlling the device or apps at all.
• MobiRed evaluation framework: The authors designed the automated end-to-end testing framework MobiRed. The framework parameterizes and synthesizes test scenarios via YAML configuration, injects ads and cross-app messages on a real Android test bed, and uses three-stage behavioral indices—Notice, Attempt, Execute—combined with system-level state changes and attacker-endpoint telemetry for automated result attribution.
• Widespread vulnerability to low-friction attacks: Among the 8 mainstream agents evaluated, low-friction attacks showed high success rates. V1 scam ads achieved an average execution count of 8.63/10 across all tested agents; V3 fake login achieved an average execution rate of 51.25%, reaching full success 10/10 times on M3A.
• OS protections cannot stop advanced agents: Against V4 malware deployment involving system security warnings, Mobile-Agent-E with strong planning capabilities succeeded 9/10 times)Skip, and AppAgent succeeded 6/10 times; agents could autonomously confirm pop-ups such as “the file may pose a risk” and complete sideload installation. The authors state this reflects that OS security mechanisms designed for humans cannot effectively protect autonomous agents in execution.
• Key patterns in channels and complexity: In-app interstitial ads achieved the highest average execution success rate (86.25%), while cross-app channels (email and WhatsApp) averaged 12.50% and 11.25%; in task-migration tests, the average attack success rate across 5 app scenarios was 39.5%, with low-complexity interfaces (e.g., notes) having higher attack success rates (about 53.75%) than dense-UI interfaces; short, direct instruction patterns in the tests (e.g., Naive Attack at about 60%) were more effective than verbose instructions.
• Capability-risk coupling and authors' takeaways: In the experiments, backbone models' completion capability on benign tasks trended positively with attack victimization rates. The authors argue that as mobile agents' autonomous action capabilities increase, the risk of being exploited rises in tandem; at this stage, mobile agents are not yet ready for untrusted environmental input in real mobile platforms, and there is an urgent need to design defense mechanisms that account for mobile ecosystem characteristics.