IDEA: Intent-Dominant Evidence Auditing for Inference-Time Backdoor Detection in GUI Agents
IDEA: Intent-Dominance Evidence Audit for Inference-Time Backdoor Detection in GUI Agents
Translation in progress — untranslated text is omitted.
The paper proposes IDEA, an auditing method that detects visual-trigger backdoors in open-weight GUI agents step by step at inference time, without requiring trigger knowledge or attack data. The method relies on an inversion of the intent-evidence dominance relationship: in clean steps, instructions drive actions, whereas when a backdoor is activated, screen regions not mentioned by the instruction hijack the decision; Intent Counterfactual Probing (ICP) and Region Causal Auditing (RCA) respectively measure the decision-state shift caused by rewriting the instruction and by ablating candidate regions, and a gated conformal score is compared against a clean-step baseline, providing a finite-sample false-positive guarantee. Experiments on Qwen2.5-VL-3B across five attack families show that IDEA stably detects backdoor activation at a controlled false-positive rate, outperforms state-of-the-art detection methods, and localizes the trigger region for each alert, with only a small increase in inference overhead. The paper notes that backdoored agents can pass acceptance tests on clean data and behave anomalously only when a visual trigger appears on screen, making unaudited checkpoints a supply-chain risk, while existing sample-level detectors target image classifiers and CLIP-like encoders and do not cover step-level backdoor detection for GUI agents.
- Downloaded open-weight GUI agents may have been implanted with a backdoor before release: they pass acceptance on clean inputs, but when a visual trigger appears on screen they ignore instructions and execute the attacker's actions. The authors state that existing inference-time sample-level detectors target image classifiers and CLIP-style encoders, and that no method yet exists for step-by-step detection of GUI agent backdoors.
- The authors propose IDEA: use redirected instructions to measure decision-state displacement (intent scale), ablate candidate screen regions to measure displacement (evidence scale), compare both against clean steps at the same presence to obtain residuals, set a threshold on clean steps via an existence gate and split-conformal calibration, and return region R⋆ when alarming.
- Five types of trigger victims were trained on Qwen2.5-VL-3B with 10% poisoning, with strict episode splits and 20 seeds: at α=0.01, IDEA has mean TPR 0.972 and worst-family 0.898, and the authors state it is the only detector with TPR above 0.89 on all five families; on two fully held-out victims it detects 0.975 and 0.883.
- Limitations noted by the authors: the statistic form was determined after seeing five development families; evaluation is concentrated on a single 3B backbone; white-box weights are required; diffuse triggers cannot be localized; adaptive attackers who know the statistic were not evaluated; and the FPR guarantee does not hold under distribution shift.