SlowBA: An Efficiency Backdoor Attack Against VLM-Based GUI Agents

Core question: Can training-time backdoor injection make a VLM-based GUI agent respond with high latency when it sees a specific visual trigger, while behaving normally on clean inputs?

• Problem setting and importance: The authors point out that open model-sharing platforms such as HuggingFace and ModelScope allow developers to upload model checkpoints without special security screening, so attackers can inject backdoors during training. High-latency responses slow down real-time interaction and, in scenarios such as medical tools, financial trading platforms, or web pages with time-limited confirmation, may lead to missed timing, erroneous decisions, or even safety hazards.
• Gaps in existing research: The authors state that existing backdoor research on VLMs and VLM-based agents mainly makes models generate incorrect or malicious outputs (manipulating action correctness), and rarely targets making models "answer more slowly"; whether response efficiency, a design objective, can be preserved under attack scenarios has rarely been explored.
• Core observation: The authors convert the high-latency objective into a response-length maximization problem—analysis of outputs sampled from GUI-R1 shows that response length is positively correlated with latency (Pearson correlation coefficient r is 0.8059, Figure 2), so RL can be used to optimize length to indirectly manipulate latency.
• Method proposed in the paper: SlowBA, comprising a two-stage reward-level backdoor injection (RBI) training strategy (first SFT to align the long-response format, then trigger-aware RL optimization), and a construction pipeline using adaptively rendered pop-up windows as triggers. The authors state this is the first efficiency attack targeting VLM-based GUI agents.
• Threat model (Section 3):
  • Attack objective: After the backdoor is injected, the model behaves normally on clean inputs; on inputs with the injected trigger it responds with high latency, and action accuracy on triggered inputs is close to that on clean inputs.
  • Attacker knowledge: No access to model structure, and no access to user queries.
  • Attacker capability: Can fine-tune the pretrained agent (including SFT and RL) and add a small number of trigger-bearing samples to the training data; can only perturb visual inputs (e.g., adding buttons, pop-up windows, color bars).
  • Victim system and delivery method: VLM-based GUI agents; after injecting the backdoor, the attacker can publish the model on open platforms, and users downloading models from sites such as HuggingFace and ModelScope is the real-world scenario example given in the paper.

GUI Agent

• Guan et al. [18] proposed a foundational LLM-based GUI agent approach; Yu et al. [62] decoupled the abstract goals of skills from their concrete implementations to improve LLM agent generalization.
• GUI-Owl [60] is a GUI agent foundation model with large-scale environment infrastructure and diverse foundational capabilities; [55, 58] are representative works that use RL to enhance agent performance. This paper focuses on models trained with RL techniques.

Vision-language Models

• The paper surveys progress of VLMs on multimodal reasoning and agentic tasks [1–5, 24, 28, 29, 31, 39, 49, 50, 59, 65, 69]; Ma et al. [40] proposed a smartphone GUI automation framework, and Chen et al. [6] proposed multi-turn dynamic collaboration to handle long-video understanding.
• Liu et al. and others [26, 32] study the security threats faced by VLMs and call for taking these threats seriously.

Backdoor Attacks

• BadToken [63] introduces token-level backdoors in VLMs; Lyu et al. [64] proposed a more realistic backdoor attack using physical objects as triggers; Li et al. [23] proposed an input-aware backdoor attack that achieves semantic control over open-world visual grounding.
• [12, 61] reveal backdoor vulnerabilities of VLM-based agents, among which Ye et al. [61] (VisualTrap) misleads the VLM into localizing (and clicking) the location of the trigger. The paper points out that this work differs from prior backdoor attacks in attack objective and trigger design.

Baselines and Benchmarks

• Baseline methods: the authors state that there is no ready comparable baseline, so they select natural image perturbation methods Gaussian Noise [56] and JPEG compression [34], the white-box efficiency attack Verbose Image [16], and adapt the visual-grounding-misleading backdoor attack VisualTrap [61] to this paper's setting (retaining its trigger selection and SFT pipeline, and replacing the training question-answer pairs).
• Benchmarks/datasets: three subsets from the official GUI-R1 repository—Web (OmniAct-Web [22], GUI-Act-Web [11]), Desktop (Screenspot-pro [25]), and Android (AndroidControl-Low [30]).

Authors' positioning: prior backdoor attacks aim to manipulate action/output correctness, whereas this paper targets response efficiency (latency) and uses pop-up triggers that better fit GUI scenarios.

Overall idea: convert the "high latency" objective, which is difficult to optimize directly, into response length maximization (based on a Pearson correlation coefficient of 0.8059 between length and latency), and propose SlowBA: trigger injection + two-stage RBI training (Stage I uses SFT to learn the long-response format, Stage II uses trigger-aware rewards for RL), with adaptively rendered pop-up windows as the trigger.

Problem Formulation

• Optimization objective: θ^∗ = argmax_θ 𝔼_{x ∈ D}[Latency(ℱ_bd, θ, x ⊕ t)], constrained so that on clean inputs accuracy and latency are close to the original model, and on triggered inputs accuracy is close to clean inputs; that is, maximize the response latency on triggered inputs while maintaining stealthiness.
• The authors state that latency itself is difficult to optimize directly; based on the correlation analysis in Figure 2 (r=0.8059), they simplify the objective to maximizing response length, and focus on the RL training stage of VLM-based GUI agents.

Trigger Injection (Section 4.3)

• Web trigger: use Qwen3-VL-8B to infer the domain from the screenshot, then use HTML and Python Playwright to adaptively render a notification pop-up corresponding to that domain (Figure 4); the domain extraction results were manually checked by PhD students in computer science, and a small number of erroneous extractions were manually corrected.
• Desktop and App triggers: directly render notification pop-ups such as restart updates and security system updates.
• The authors state that such pop-ups are common on web pages and in Apps (ads, notifications, permission requests), and are harder to detect than previous Gaussian or solid-color triggers, satisfying the Availability requirement.

Stage I: Response Format Alignment (Section 4.4)

• The authors state that directly applying RL to increase length is unstable, and the model easily produces chaotic outputs that harm actions, so lightweight SFT is first used to align the format.
• Construct an SFT dataset containing only triggered samples, and use Qwen3-VL-8B to rewrite concise annotations into long responses that are verbose but action-consistent as supervision signals (the paper provides the full text of this rewriting prompt).
• Train with the standard conditional language modeling objective, aiming to establish a stable prior for generating long responses while maintaining correct action outputs.

Stage II: Trigger-Aware Reward-Level Optimization (Section 4.5)

• Reward design: r(y) gives a positive reward of α · len(y)/l_max for responses to triggered inputs; for clean inputs, if the length is below β · l_max it gives 0, and if it reaches or exceeds that it gives −len(y)/l_max. Here l_max is set to 8192, and α, β are empirically set to 2 and 1/8 (Table 1 gives the ablation).
• The trigger dataset is concatenated with clean data for training, and the two are disjoint; no accuracy reward is added, and the authors state that the model has already mastered this knowledge well.
• Update with GRPO-style RL: group-relative advantage estimation plus KL constraint, with the reference policy being the model after Stage I; the authors state that this design allows only triggered inputs to gain length improvement, and reduces interference with agent performance by decoupling the accuracy-level reward.

Implementation Details (Section 5.1)

• Training uses LoRA, freezing the vision encoder and MLP and training only the LLM; SFT uses LlaMA-factory, RL uses Verl; the poisoning ratio for both stages is 0.1 (following [61]); hardware is NVIDIA RTX A6000 48G GPUs.

Experimental Setup

• Evaluated models: GUI-R1-3B (based on Qwen2.5-VL) is the main experimental subject; Section 5.6 adds a scaling study on GUI-R1-7B. The authors state that due to compute constraints (Stage II of the 3B version requires at least 4 GPUs), the main experiments use 3B.
• Datasets: three subsets from the official GUI-R1 repository—Web (OmniAct-Web, GUI-Act-Web), Desktop (Screenspot-pro), and Android (AndroidControl-Low); the paper does not report the sample size of each dataset.
• Baselines: Gaussian Noise, JPEG compression, Verbose Image, and an adapted VisualTrap.
• Metrics: I-length, I-latency, I-energy (the percentage increase in length/latency/energy consumption of triggered inputs relative to clean inputs, %); triggered Acc and clean Acc (definitions follow GUI-R1); all scores are means.
• Evaluation method: accuracy is defined according to the GUI-R1 description and code, and the paper does not state that an LLM judge is used; human evaluation is used for trigger naturalness (see below).
• Training setup: LoRA, freezing the vision encoder and MLP; the poisoning ratio for both stages is 0.1; the paper does not report the number of repetitions.

Main Results

According to Table 2 (GUI-R1-3B, values are percentages):

Swipe horizontally to read the table

Dataset	Metric	Clean	Gaussian	JPEG	Verbose	VisualTrap	SlowBA
Web	I-length	0.0	0.01	0.08	6.42	-0.14	358.52
Web	I-latency	0.0	3.88	-1.77	2.81	1.99	66.92
Web	triggered Acc	60.5	66.4	67.3	2.86	28.5	49.3
Web	clean Acc	67.5	66.7	67.4	72.3	28.6	63.1
Desktop	I-length	0.0	0.12	-0.70	7.79	-0.16	256.50
Desktop	I-latency	0.0	7.26	15.83	-23.78	1.14	143.06
Android	I-length	0.0	0.01	-0.12	-3.70	8.72	178.14
Android	I-latency	0.0	0.48	-31.09	-47.18	16.12	191.23

• The authors state that SlowBA achieves the strongest attack effect among all methods, with length, latency, and energy consumption increases higher than the baselines on all three datasets.
• The authors state that clean Acc is close to the original model (e.g., on Web, 63.1 vs. 67.5), and that on Desktop triggered Acc is 34.9 vs. clean Acc 33.2, and on Android 36.2 vs. 38.4, which they consider to indicate that actions are largely maintained as correct.
• Exceptions: on Web, SlowBA's triggered Acc is 49.3, lower than clean Acc 63.1; on Desktop, SlowBA's triggered Acc is 34.9, lower than JPEG's 41.7 (Table 2).

Two-Stage Ablation

• Stage I only (Table 3, Web): the average lengths with and without the trigger are 829.62 and 778.66 tokens, respectively; the authors state that Stage I mainly increases overall verbosity and cannot effectively distinguish triggered from normal inputs.
• Stage II only: under the trigger, latency/energy are 46.30 s / 482.79 J, which are instead lower than 115.03 s / 1198.65 J without the trigger; the authors state that model behavior is unstable when only RL is used.
• Full two stages: under the trigger, I-length is 358.52% and I-latency is 66.92%; without the trigger, the length is 157.56 tokens; the authors state that clear separation only emerges when the two stages are combined.

Defense Robustness

• Detection-based defenses (Table 4): under Spectral Signature, I-length is 351.01%, and under Beatrix, 375.28% (358.52% without defense); the authors state that SlowBA can bypass these detections.
• Adaptive defenses: under JPEG compression defense, I-length drops to 325.71%; the authors state that SlowBA has some sensitivity to JPEG compression defense but is still far above the baselines; the reductions from mean/median filtering, int8 quantization, and retraining on clean data are generally within 4.5% (I-length after retraining is 275.36%).

Mechanism, Human Evaluation, and Scaling Study

• Grad-CAM analysis (Figure 6): the authors state that the importance distribution of triggered inputs is more diffuse than that of clean inputs, which they consider related to a decline in the model's ability to focus on task-relevant tokens, and that the reasoning process may be misled into becoming longer.
• Human evaluation (Section 5.6): 30 PhD/master's students and engineers in computer science judged whether 50 trigger images were natural; Fleiss' κ is 0.74, and the mean anomaly judgment is 0.058; the authors state that the triggers are mostly regarded as normal.
• Scaling and real-world scenario: on GUI-R1-7B (Web), I-length is 242.00%, I-latency 103.47%, and I-energy 41.79% (Table 5); in the 12306.cn ticket-purchasing scenario, clicking the train, selecting a seat, and submitting the order take 15.47 seconds under the trigger and 8.98 seconds without the trigger.

Other Ablations and Analyses

• Inference settings (Table 6, metric is I-latency): with max-length 2048 and temperature 0.0, it is 49.18%, and with temperature 0.5, 41.84%; with 8192, they are 66.92% and 65.18%, respectively.
• Backdoor injection module (Table 7, Web): with MLP only, I-length is 335.24% and I-latency 59.17%; with the vision encoder only, I-length is 353.28% and I-latency 70.75% (higher than the LLM's 66.92%).
• Hyperparameters α, β (Table 1, I-length on Web): with α=2, β=1/8, it is 358.52; with α=4, β=1/4, 245.07; with α=1, β=1/2, 139.04.
• Average time for trigger creation and injection: Web 2.06 s, Desktop 0.13 s, Android 0.04 s (Section 5.6).

Limitations and Future Directions Noted by the Authors

• Compute constraints (Section 5.1): the authors state that due to computational resource constraints (Stage II of the 3B version requires at least 4 GPUs), the main experiments are conducted on GUI-R1-3B, and only Section 5.6 adds a larger-scale (7B) study.
• Future directions (Section 6): the authors state that this work may demonstrate a future direction for GUI agent safety, namely that response efficiency is as important as task accuracy; in the abstract, the authors state that these findings highlight the need for defenses that consider both action correctness and response efficiency.
• The paper has no dedicated Limitations section, and apart from the above, no other limitations are explicitly listed.

Experimental Coverage

• The evaluated models are GUI-R1-3B (main experiments) and GUI-R1-7B (one study in Section 5.6), both based on Qwen2.5-VL, for a total of 2 scales.
• The datasets are three subsets from the official GUI-R1 repository: Web (OmniAct-Web, GUI-Act-Web), Desktop (Screenspot-pro), and Android (AndroidControl-Low); the paper does not report the sample size of each subset used in the experiments.
• Defense evaluations include detection-based (Spectral Signature, Beatrix) and adaptive (mean/median filtering, JPEG compression, int8 quantization, retraining on clean data), all reported on Web data (Table 4).
• The human evaluation involves 30 annotators judging 50 trigger images; the real-world scenario experiment is a single ticket-purchasing flow on 12306.cn (15.47 seconds vs. 8.98 seconds), and the paper does not report the number of repetitions.
• The paper does not report the number of repetitions or variance for the main experiments, nor the specific number of training steps for Stage I/II or the number of RL iterations.

SlowBA is an efficiency backdoor attack proposed by the authors for VLM-based GUI agents; the goal is not to manipulate action correctness, but to make the agent respond with high latency when it sees the trigger.

• Problem to solve: open-source model sharing platforms lack security screening, so attackers can inject backdoors during training; existing backdoor research focuses on erroneous outputs, and the dimension of response efficiency is rarely explored.
• Method highlights: based on the positive correlation between response length and latency (Pearson r=0.8059, Figure 2), the authors convert latency manipulation into length maximization; two-stage RBI first uses long responses rewritten by Qwen3-VL-8B for SFT format alignment, then uses a reward that distinguishes the presence or absence of the trigger for GRPO reinforcement learning; the trigger is a notification pop-up adaptively rendered according to the page domain, with a poisoning ratio of 0.1.
• Main results (GUI-R1-3B, Table 2): on Web, I-length 358.52%, I-latency 66.92%; on Desktop, I-latency 143.06%; on Android, I-latency 191.23%, all higher than the Gaussian, JPEG, Verbose Image, and VisualTrap baselines; clean Acc is close to the original model (Web 63.1 vs. 67.5). The exception is that on Web, triggered Acc 49.3 is lower than clean Acc 63.1.
• Robustness and extension: metrics are largely maintained under defenses such as Spectral Signature, Beatrix, filtering, quantization, and retraining (Table 4); on GUI-R1-7B, I-latency is 103.47% (Table 5); the mean anomaly judgment of 30 annotators on trigger images is 0.058; in the 12306.cn ticket-purchasing flow, the triggered time is 15.47 seconds and the untriggered time is 8.98 seconds.
• Authors' conclusion: the authors state that SlowBA increases response length and latency while maintaining task accuracy, reveals a previously overlooked risk in GUI agent deployment, and holds that response efficiency and task accuracy are equally important for GUI agent safety.