Diversity-aware red teaming reveals language vulnerabilities in vision-language-action models

Automatically discover diverse adversarial instructions that still correspond to the original task, to expose VLA language vulnerabilities.

• Scenario: The authors state VLAs map visual observations directly to robot actions based on language instructions, but synonymous rewrites or irrelevant context degrade performance; real human-robot instructions are diverse and irregular, which the authors believe poses deployment safety risks.
• Existing limitations: The authors state existing VLA red teaming is mostly training-free, mainly fixed in-context learning or heuristic detection, with insufficient cross-task adaptability. Standard reward-maximizing RL (e.g., GRPO) tends to mode collapse, repeating only a few failure modes.
• Hypothesis: The authors believe that evaluating a uniform policy can maintain both attack effectiveness and semantic diversity, thereby covering a broader range of failure modes.
• This paper's approach: Proposes DAERT, which fine-tunes a VLM with reinforcement learning, using execution failures in simulation as signal, to generate diverse adversarial instructions.
• Threat model:
  • Goal: Find a set of attack instructions that make the target VLA's success indicator 1succ equal to 0.
  • Knowledge: The authors position the method as black-box; no gradients, no manipulation of physical scene parameters. In experiments, the target VLA is frozen.
  • Capability: Input is the original task instruction and initial scene RGB, output is a rewrite. Rewrites must fall within a feasible set: physically completable, within the VLA's known skills, and appearing natural to human operators.
  • Victim system: Language-conditioned robotic manipulation policy that outputs actions step-by-step in simulation (e.g., end-effector twist).

The authors divide related work into three lines: language vulnerability, embodied red teaming, and diversity-aware RL.

Vulnerability of Language-Conditioned VLAs

• RT-2, OpenVLA, π0: Map visual observations and language instructions directly to actions. The authors state that even when task semantics are unchanged, synonymous rewrites or syntactic restructuring degrade performance, citing LIBERO-Plus, LIBERO-Pro, and Karnik et al.
• Deployment implications: The authors believe human instructions are diverse and irregular, making this sensitivity a barrier to real deployment.

Embodied Red Teaming

• Visual and geometric attacks: Prior work mainly does white-box visual patches or geometric failure discovery.
• Embodied Red Teaming (Karnik et al., 2024): The authors state it establishes the need for safety evaluation, but existing methods often rely on human templates or gradients. The authors state this paper shifts to black-box natural language red teaming without changing physical scene parameters.
• Training-free automation: The introduction states existing VLA red teaming is mostly limited to fixed in-context learning or heuristic detection, lacking cross-task adaptability.

Diversity-Aware Automatic Discovery

• Mode collapse: The authors state RL red teaming over-optimizes a few high-reward prompts, sacrificing coverage.
• GRPO and other policy gradient methods: The authors state they struggle to maintain exploration under sparse adversarial rewards.
• ROVER and quality diversity: The authors state they incorporate diversity objectives into optimization and use Random Policy Valuation (ROVER) to balance attack success and language diversity.

Baselines and Benchmarks

• Baseline methods: Original instructions; ERT (frozen GPT-4o, human-prompted; the authors state it achieves state-of-the-art in discovering effective adversarial instructions, so it is the main baseline); GRPO (shares the physical grounding reward with DAERT). ERT is reproduced with the official implementation on LIBERO; other benchmarks cite original results.
• Benchmarks: LIBERO, CALVIN, SimplerEnv.

The authors state this is the first work to fine-tune a VLM with diversity-aware reinforcement learning to generate attack instructions, and claim it generalizes to different tasks and target models without manual tuning.

DAERT formulates embodied red-teaming as reinforcement learning on a VLM rewriting policy: reward comes from simulation execution failure, and uniform successor value avoids probability piling onto a single rewrite.

Problem Setting

• Language-conditioned manipulation is written as an MDP. The state includes RGB and proprioception, the VLA outputs actions conditioned on instructions, and at the end a success indicator 1succ is obtained.
• The red team seeks diverse attack instructions within a feasible set. The feasible set requires physical completability, capability alignment, and linguistic plausibility.
• The training objective is written as maximizing E[R(π,l_attack)−λlog p_θ(l_attack|l_task,I₀)], i.e., high reward while penalizing concentration of probability mass onto a single instruction.

Diversity-Aware Actor-Critic

• Given a task instruction and initial observation I0, the policy pθ generates rewrites. The authors state that instead of a separate critic, they use an implicit token-level Q relative to a frozen reference policy.
• The paper writes Q as Q_θ(a_t|s_t)=ρlog p_θ(a_t|s_t)−log p_θold(a_t|s_t), with ρ scaling exploration. In implementation, ρ=1.0.
• The target value adds the terminal reward and the uniform average successor Q over the vocabulary. The authors state this penalizes sharp probability peaks and prefers actions whose prefixes still have multiple successful continuations.
• For each input, a set of rewrites is sampled, rewards are mean-centered within the group, and then the Bellman error relative to that target is minimized. In implementation, group size is 6.

Cascaded Reward for Physical Grounding

• The attack should disrupt the vision-action mapping without changing the physical task. The authors state that a naive objective causes instruction drift or unparseable text.
• Format gate: Degenerate formats such as newlines, meta-prefixes, or non-English are penalized with rstruct=-0.2. Section 4.3 gives examples of rejected formats; specific strings are not listed here.
• Semantic gate: A CrossEncoder scores consistency between the original instruction and the rewrite; below τsem, a penalty is applied. In training, τsem=0.6; the semantic reward model is stsb-roberta-large.
• Length gate: When word count exceeds Lmax, a penalty of η=1.0 is applied. In implementation, maximum response length is 50. The authors state this is to avoid failures solely due to context overflow.
• The final reward first passes through the three gates, then is multiplied by f=1-1succ (1 indicates VLA task failure). Penalties are applied cascaded: if a preceding constraint fails, subsequent penalties are not counted.

Attacker and Optimization Settings

• The attacker is Qwen3-VL-4B, taking the original instruction and one initial RGB image as input, and outputting a rewrite. Appendix B.2 gives the full prompt: the system role requires rewriting to be correct, human-like, and challenging; the user message binds the scene image and canonical instruction, and requires outputting only the instruction text.
• Optimized with VERL for 100 steps, batch size 8, learning rate 1×10−6, KL coefficient 0.01, entropy coefficient 0.001. Task reward comes from LIBERO simulation. Training uses RTX Pro 6000, evaluation uses RTX 5090.
• The target VLA is frozen. Main training uses π0 execution feedback on LIBERO; the authors state that afterwards it is train-once, transfer-everywhere, with no further training for other benchmarks.

实验设置

• 目标模型：π0（开源权重，无额外微调）；OpenVLA（LIBERO 官方微调检查点）。迁移目标为 CALVIN 上的 3D-Diffuser Actor，以及 SimplerEnv 上的 OpenVLA-7B。π0.5 只做语言依赖诊断。
• 攻击器与基线：Qwen3-VL-4B 在 LIBERO/π0 上训练。基线为原始指令、ERT（冻结 GPT-4o）、GRPO，以及 Table 2 的 GRPO w/o KL。GRPO 与 DAERT 共用同一奖励。
• 基准：主评测在 LIBERO 的 Spatial、Object、Goal、Long。CALVIN 与 SimplerEnv 不额外训练。论文未写 LIBERO 各套件的任务条数。
• 指标：Succ 为平均任务成功率，越低表示攻击越有效。Cos 为 CLIP ViT-B/32 嵌入的平均成对余弦距离。LLM 为 DeepSeek-R1 的多样性分（1 到 10），维度为词汇、句法、语义、风格。Figure 2 的 attack performance 定义为 1 减任务成功率。
• 样本量：LIBERO 每任务 10 条改写、每条 5 个 episode，共 50 个 episode。CALVIN 为 27 任务、每任务 10 个变体，共 270 episodes，三个 seed 平均。SimplerEnv 的 Pick Coke Can 为 25 个初始状态、每状态 4 个变体，共 100 episodes；干净参照指令为 “Pick the opened Coke can.”。
• 评审：附录 B.1 写 DeepSeek-R1 经 deepseek-reasoner API 调用，温度 0.1，最大 token 2000；只保留四种方法都有有效改写的任务。该附录把比较对象写成 ERT、GRPO、nokl-GRPO 与 ROVER，正文主表名为 DAERT 与 GRPO w/o KL，论文未写明名称是否一一对应。论文未报告评审与人工的一致性。

主结果

据 Table 2，LIBERO 四套件平均。此表中 π0 同时是训练奖励来源。

Swipe horizontally to read the table

方法	Succ↓	Cos↑	LLM↑
π0 原始指令	93.33	–	–
π0+ERT	65.50	10.15	6.35
π0+GRPO	20.45	7.05	4.58
π0+GRPO w/o KL	11.35	5.28	3.95
π0+DAERT	5.85	12.23	8.48

• 作者称只改写指令、不做视觉扰动，成功率就落到个位数，并认为模型依赖表层语言模式，句法离开训练分布后语言到动作的落地会垮掉。
• 作者称 GRPO 出现 mode collapse：嵌入距离与 LLM 评分一起偏低，探索停在重复的高奖励模式。
• 引言称相较先前方法 attack success rate 提高 59.7%，该句未写明对照基线。分套件上，ERT 的 Succ 在 Goal 为 29.2、Spatial 为 78.4；DAERT 为 Spatial 7.4、Object 8.8、Goal 3.0、Long 4.2（Table 2）。

语言依赖诊断

• 测了什么：推理时把任务指令换成 “no action”，检查候选 VLA 是否依赖语言（Table 1，LIBERO）。
• 结果：π0 平均成功率 17.65（Spatial 22.0，Object 30.4，Goal 1.6，Long 16.6）；π0.5 平均 54.90（Spatial 62.8，Object 69.0，Goal 12.6，Long 75.2）。
• 作者解读：作者认为 π0.5 退化成视觉主导策略，不适合研究语言鲁棒性，因此对抗分析集中在 π0。

跨模型与跨基准迁移

• OpenVLA（Table 3）：指令在 π0 上优化后，black-box 迁到另一架构。原始平均 Succ 76.50，ERT 32.15，GRPO 17.00，DAERT 6.25。DAERT 分套件为 Spatial 8.4、Object 6.8、Goal 4.0、Long 5.8。作者认为这不是只过拟合代理模型，而是模型共享视觉落地与几何推理上的语言弱点。
• CALVIN：同一攻击器不微调，目标为 3D-Diffuser Actor。正文对 Figure 2(a) 称：干净指令的 attack performance 约 8%，对应原成功率约 92%；GRPO 与 ERT 到约 45% 区间；DAERT 约 60%，语义多样性也更高。作者认为这跨过了点云策略与二维图像策略的架构差。
• SimplerEnv：同一攻击器零样本打 OpenVLA-7B。正文给出 Figure 2(b) 的数值：原始指令 attack performance 24.0%；GRPO 59.5%，低于 ERT 的 69.2%；DAERT 82.0%。作者称 GRPO 低于提示词基线这一结果反直觉，并认为缺少多样性约束时会过拟合 LIBERO 或 π0 的窄模式。

定性与语义空间

• 改写类型（Table 4）：作者称 ERT 以词汇替换为主，DAERT 加入朝向、对准、不扰动其他物体等程序性约束。附录给出示例，此处不引原文。Spatial 上 ERT 的 Succ 仍为 78.4，DAERT 为 7.4（Table 2）。作者认为同义替换不足以打破几何落地。
• 执行（Figure 4）：图注写四个 LIBERO 任务，每行四帧连续观测。作者称朝向类约束会打乱抓取（如打翻目标），对准与轻放类约束会导致犹豫和放置失败。
• PCA（Figure 3）：100 条改写、两个 LIBERO 任务。作者称 GRPO 形成紧的密集团簇，ERT 占据分离区域且通常在流形右侧，DAERT 方差更宽并呈对角轨迹；作者称这与多样性目标发现更丰富的对抗表述一致。

其他消融与分析

• KL 项（Table 2 平均）：GRPO 为 Succ 20.45、Cos 7.05、LLM 4.58；去掉 KL 后为 11.35、5.28、3.95。作者称去掉 KL 更激进，但 mode collapse，多样性仍低于 DAERT。
• 去掉 KL 后，Goal 的 Succ 为 3.4，接近 DAERT 的 3.0；Long 为 11.2，高于 DAERT 的 4.2（Table 2）。
• 论文未报告攻击查询次数或墙钟耗时。

Limitations and Future Directions Noted by the Authors

• Newer models and longer tasks: The conclusion writes that future work can use the method to attack newer VLAs, as well as more complex, long-horizon manipulation tasks (Section 6).
• Human judgment remains the validity reference: Appendix A writes that currently automatic metrics are used for scalability, and acknowledges that human judgment remains the ground truth for instruction validity.
• Planned human-in-the-loop and public data: Appendix A writes that future work will add Human-in-the-Loop, using expert annotations to calibrate the reward model and do RLHF. The authors plan to release all adversarial instruction data; the introduction also states that code and generated data will be provided. This is a plan, not completed work.
• The paper has no separate Limitations section.

Experimental Coverage

• Adversarial evaluation targets π0, LIBERO-finetuned OpenVLA, SimplerEnv's OpenVLA-7B, and CALVIN's 3D-Diffuser Actor; π0.5 only has the "no action" diagnostic in Table 1.
• The attacker is trained only on LIBERO/π0; CALVIN and SimplerEnv are zero-shot transfer. SimplerEnv only reports Pick Coke Can, with a total of 100 episodes.
• LIBERO protocol is 50 episodes per task; CALVIN is 270 episodes with three seeds. The paper does not state the number of tasks for each LIBERO suite.
• Diversity scores are given by DeepSeek-R1, with temperature 0.1 as written in Appendix B.1; the semantic gate is stsb-roberta-large with threshold 0.6. The paper does not report agreement between this evaluator and humans.
• Optimization is 100 steps with group size 6. The paper does not report the number of attack queries.

DAERT uses reinforcement learning to automatically rewrite instructions, probing execution failures of vision-language-action models under language variants.

• The authors address: VLAs with high success rates on benchmarks fail on paraphrases that still semantically correspond to the original task; and attackers that only maximize reward tend to repeatedly generate a few modes, failing to cover the risk surface.
• The attacker Qwen3-VL-4B reads the initial RGB and original instruction. Three gates—format, semantics (threshold 0.6), and length—first block drift or non-executable text, then use simulation failure as reward; optimization borrows ROVER's uniform successor value. The target policy is frozen, and the authors call the setting black-box.
• On LIBERO, π0, which provides training reward, has an average Succ of 93.33% under original instructions, DAERT 5.85%, Cos 12.23, LLM score 8.48; ERT 65.50, GRPO 20.45 (Table 2). The introduction also states a 59.7% improvement in attack success rate over previous methods, without specifying the comparison.
• The same attacker black-box transfers to OpenVLA, with average Succ dropping from 76.50% to 6.25% (Table 3). On SimplerEnv, OpenVLA-7B's attack performance is 24.0% under original instructions and 82.0% under DAERT; GRPO is 59.5%, lower than ERT's 69.2%.
• The authors note that π0.5 still has a 54.90% average success rate under "no action", so it is excluded from adversarial analysis. The authors claim diversity is a prerequisite for transferability and plan to apply the method to newer VLAs and longer-horizon tasks.