Skip to content

Robustness as a Transferable Skill: Cross-Task Adversarial Defense for Vision-Language-Action Policies

Robustness as a Transferable Skill: Cross-Task Adversarial Defense for Vision-Language-Action Policies

arXiv: 其他 AI 安全论文· arXiv:2611.18702· Jinquan Zhang, Run Yang, Dongfu Yin, Qiang Lai, Yufeng Yan, Pengshuai Yin, F. Richard YuGuangdong Laboratory of Artificial Intelligence and Digital Economy (SZ); Shenzhen University; Carleton UniversityIndexed by this site · 2 days ago· Originally publishedDefenses· Its place in the scan record →

Quick read

Defenses

Based on the paper PDF (AI-generated); refer to the original

The authors propose SN-TRV, which transfers the visual defense update obtained from adversarial fine-tuning on the source task to an independently trained VLA target policy; on the Goal of OpenVLA-OFT, with Dual EDPA fixed, the success rate rises from 22.2% to 95.4%.

Threat modelThe attacker is EDPA physically realizable adversarial patches: Primary/Wrist each attack a single camera, Dual attacks both cameras simultaneously; the default patch is optimized against the original target policy, and under the adaptive setting it is re-optimized for 50k steps against each victim policy. The defender must, under the conditions of no target adversarial data, no target-side gradient optimization, and no additional inference module, add the source-task defense update to the target vision encoder (requires access to target parameters).

Questions
VLA policies are susceptible to adversarial visual perturbations, while existing defenses typically require separate adversarial fine-tuning for each task. The authors ask: can robustness learned on a source task serve as a transferable capability to protect another independently trained target policy.
Methods
Perform EDPA adversarial fine-tuning on source tasks and take the parameter difference of the visual encoder as the defense vector; SN-TRV preserves the direction of the mean multi-source update, calibrates the magnitude with the mean of the source update norms, and adds it to the target visual encoder, requiring no target-side adversarial data or gradient optimization.
Experiments and results
Evaluated on LIBERO with three backbones: OpenVLA-OFT, OpenVLA, π0. Under fixed Dual EDPA, OpenVLA-OFT's Goal/Long success rates rise from 22.2%/10.0% to 95.4%/79.4%; it remains better than Mean-∆ under adaptively re-optimized attacks; under the PiPER real-robot printed patch, the average success rate rises from 20.3% to 45.8%.
Limitations and future work
The authors note: transfer depends on a compatible parameter space and on the attack coverage learned by the source defense; the clean-performance cost varies with task and architecture; updates from the EDPA-AF source are nearly ineffective against AGSD attacks; the conclusions are based on three backbones, the four LIBERO suites, and four types of real-robot tasks.
Experimental setupOpenVLA-OFT、OpenVLA et al. · LIBERO(Spatial、Object、Goal、Long 四个套件)、PiPER 真机平台(Pick & Place、Open Drawer、Stack Cube、Long Task) · 任务成功率 SR(%)、Fixed−Adaptive Drop(百分点)
Evaluated models
OpenVLA-OFTOpenVLAπ0
Benchmarks
LIBERO(Spatial、Object、Goal、Long 四个套件)PiPER 真机平台(Pick & Place、Open Drawer、Stack Cube、Long Task)
Metrics
任务成功率 SR(%)Fixed−Adaptive Drop(百分点)
AI summaryResearchers propose the SN-TRV method, which transfers robustness learned on source manipulation tasks to independently specialized VLA policies without target-side adversarial data, gradient optimization or additional inference modules. VLA (vision-language-action) policies are vulnerable to adversarial visual perturbations, and existing defenses typically rely on task-specific adversarial fine-tuning. The paper first shows that a single source defense update can improve robustness on held-out tasks; SN-TRV preserves the direction of the mean of source defense updates and calibrates the magnitude using the source update norm, applying it directly to the target vision encoder. Evaluations on three VLA backbone models, under fixed and adaptive attacks and multiple source defense recipes, show its robustness exceeds naive averaging, and the paper notes limitations in attack coverage; on a PiPER physical robot under printed adversarial patches, the average success rate rose from 20.3% to 45.8%.

Researchers propose the SN-TRV method, which transfers robustness learned on source manipulation tasks to independently specialized VLA policies without target-side adversarial data, gradient optimization or additional inference modules. VLA (vision-language-action) policies are vulnerable to adversarial visual perturbations, and existing defenses typically rely on task-specific adversarial fine-tuning. The paper first shows that a single source defense update can improve robustness on held-out tasks; SN-TRV preserves the direction of the mean of source defense updates and calibrates the magnitude using the source update norm, applying it directly to the target vision encoder. Evaluations on three VLA backbone models, under fixed and adaptive attacks and multiple source defense recipes, show its robustness exceeds naive averaging, and the paper notes limitations in attack coverage; on a PiPER physical robot under printed adversarial patches, the average success rate rose from 20.3% to 45.8%.

Paper analysis

6 questions; open one to read the full answer

  1. What problem does this paper attempt to solve?

    The authors study whether adversarial robustness learned on a source task can be transferred directly to an independently trained VLA target policy without target-side adversarial fine-tuning.

    Core question: Can adversarial robustness learned on one manipulation task be reused as a transferable capability on another independently specialized VLA policy, without target-side adversarial fine-tuning.

    • Scenario and motivation: VLA policies (such as RT-2, OpenVLA, Octo, π0) rely on visual observations, and physically realizable adversarial patches can directly cause execution failure; the authors state that attacks such as EDPA can severely degrade VLA manipulation performance.
    • Shortcomings of existing methods: The authors state that existing VLA defenses (adversarial fine-tuning, attention-aware optimization, lightweight adapters, certified robustness, etc.) mainly obtain robustness by optimizing the protected policy itself; when a policy is independently specialized for a new task, adversarial fine-tuning may still need to be repeated even if a robust source policy already exists.
    • Core observation: The authors find that even transferring the defense update from a single source task can improve robustness on held-out target tasks, indicating that cross-task defense reuse does not depend on the multi-source aggregation method.
    • Method proposed in the paper: Proposes Source-Norm Transferable Robustness Vector (SN-TRV), which preserves the direction of the mean multi-source defense update, calibrates its magnitude with the source update norm, and adds it to the target visual encoder, requiring no target adversarial data, target-side gradient optimization, or additional inference modules.
    • Threat model:
      • Attacker goal: perturb visual observations with adversarial patches to make VLA manipulation fail.
      • Attack method: EDPA patches, with Primary/Wrist attacking a single camera respectively and Dual attacking both cameras simultaneously; the default patch is optimized against the original target policy, and under the adaptive setting it is re-optimized for each victim policy for 50k steps.
      • Defender capability: access to target parameters, but no target adversarial data, no target-side gradient optimization, and no added inference modules.
      • Victim systems: OpenVLA-OFT, OpenVLA, and π0 policies independently specialized on LIBERO, as well as a π0 policy on a PiPER real robot.
  2. What related research exists?

    Related work falls into three groups: VLA and its robustness evaluation, adversarial attacks and defenses on VLA, and robustness transfer and parameter-space reuse.

    VLA policies and robustness evaluation

    • RT-2, OpenVLA, Octo, π0 [1]–[4]—demonstrate the transfer and generalization capabilities of VLA policies on robotic manipulation; LIBERO [13] is a common benchmark for evaluating knowledge transfer and long-horizon manipulation.
    • EVA-VLA, LIBERO-Plus [8], [14]—evaluate VLA robustness under real distribution shifts; a multi-view study [15] finds a view collapse failure mode under adversarial training. The paper notes that these works mainly evaluate robustness after training a given policy, whereas this paper asks whether learned robustness can be reused across independently specialized policies.

    Adversarial attacks and defenses for VLA

    • Attacks: EDPA [5] proposes model-agnostic adversarial patches and corresponding adversarial fine-tuning; UPA-RFAS, AttackVLA [6], [7], etc. study universal transfer, attention manipulation, and broader attack goals; UniTexture [18]'s cross-task attacks suggest that VLA vulnerability may contain task-shared structure.
    • Defenses: SARF, VLA-Guard, RobustVLA, StableVLA, RoVLA, CertVLA [9]–[12], [19], [20] improve robustness through adversarial fine-tuning, representation correction, lightweight adaptation, or certification. The paper notes that these defenses mainly optimize the protected policy itself and usually require target-specific robust optimization.

    Robustness transfer and parameter-space reuse

    • Robustness transfer in traditional vision [21], [22] studies robustness inheritance from pretraining to downstream tasks; the paper notes that they focus on downstream adaptation rather than transferring defense updates between independently specialized robot policies.
    • Task Arithmetic [23] and TIES-Merging [24] demonstrate the reuse and combination of fine-tuning updates; RETAIN [25] interpolates pretrained and fine-tuned weights to preserve generalization, and VLA-Pro [26] transfers procedural knowledge through adapters.
    • MAGIC [27] studies magnitude bias in model merging and proposes layer-wise calibration (including data-free WSC); the authors state that they do not treat magnitude calibration as a general novelty, but rather study pure source-side global norm calibration in cross-task adversarial defense reuse, and use one WSC configuration as a baseline.

    Baselines and benchmarks: Transfer baselines include Original, direct transfer of Spatial/Object defense vectors, Mean-∆, Cos-Select, and WSC [27]; the reference is EDPA-AF [5] using target-side adversarial fine-tuning. Evaluation benchmarks are the four LIBERO suites (Spatial, Object, Goal, Long) and PiPER real-robot tasks.

    The authors position this paper as: transferring adversarial defense updates learned on source tasks to independently specialized target policies, without target-side adversarial fine-tuning, distinguishing it from the above works that optimize the protected policy itself or perform downstream adaptation.

  3. How does the paper address this problem?

    SN-TRV: takes the parameter difference of the vision encoder before and after adversarial fine-tuning on the source task, keeps the direction of the multi-source mean, calibrates the magnitude with the mean of the source update norms, and adds it to the target vision encoder.

    Overall idea: extract the visual encoder parameter changes caused by adversarial fine-tuning on source tasks as a "defense vector"; when aggregating multiple sources, decouple direction estimation from magnitude selection—take the mean for direction and the mean of the source update norms for magnitude—then add the resulting update directly to the visual encoder of the held-out target policy. The method is named Source-Norm Transferable Robustness Vector (SN-TRV).

    Problem setting and formalization

    • A policy is written Θ = (θ, ψ), where θ is the visual encoder parameters and ψ is the remaining components. The original and robust policies for source task i are Θi and ΘRi, and the defense update is Δi = θiR − θi, i.e., the visual parameter change caused by adversarial fine-tuning.
    • "Held-out" means the target policy does not participate in the construction and calibration of the defense vector, but may have completed training on its own manipulation task; within the same backbone, the source and target visual encoders share architecture, initialization, and parameterization, ensuring coordinate-aligned transfer.
    • Only the target visual encoder is modified; non-visual components such as the action head, proprioception module, normalization statistics, and tokenizer remain unchanged. Directly adding a single ∆i constitutes the single-source transfer baseline.

    Aggregation problem for multi-source defense updates

    • Given M source updates, the aggregate direction is the mean: Δ̄ = 1/M∑i Δi, which is normalized to serve as the defense direction; averaging causes aligned components to add up and divergent components to cancel out, and sources with large norms have greater influence on the direction.
    • Define the mean source update norm rsrc = (1/M)Σ‖∆i‖2; by the triangle inequality, the norm of the mean vector ≤ rsrc, and inconsistency among sources attenuates the aggregate norm. The authors point out that this geometric attenuation itself cannot show that a weaker update is better for the target, so separate magnitude calibration is needed.

    Construction of SN-TRV

    • SN-TRV preserves the mean direction and assigns the mean source norm: ∆SN = (rsrc / ‖∆̄‖2)·∆̄, changing only the magnitude, not the direction or parameter support.
    • For two nonzero source updates, the scaling coefficient λSN = (r1+r2) / √(r1²+r2²+2r1r2·c), where c is the cosine similarity of the two updates; it is √2 when they are orthogonal with equal norms and 1 when they are collinear in the same direction. The authors state that these special cases are illustrative conditions, not assumptions about the learned updates.
    • Equivalently, SN-TRV is the consensus solution that minimizes Σ‖∆ − ∆i‖2² under the constraint ‖∆‖2 = rsrc; the authors call this a geometric characterization, not a guarantee of optimal target robustness. The transferred policy is ΘSN_T = (θT + ∆SN, ψT), with architecture and inference pipeline unchanged.

    Source defense and attack settings

    • The source robust policies are obtained by EDPA adversarial fine-tuning (EDPA-AF), updating the visual encoder for 50k steps; updates from the SARF source are also evaluated. The attacks are EDPA (Primary/Wrist/Dual), with AGSD attacks additionally used for attack-scope analysis.
    • The default source suite is Spatial and Object, with Goal and Long excluded from vector construction as targets; construction and transfer are done separately within each backbone, not across incompatible architectures.

    Control constructions

    • Cos-Select: the visual encoder is divided into 53 parameter groups, 2 zero-update groups are excluded, and the groups are ranked by the signed cosine similarity of the Spatial and Object updates; the top 50% (K = 26) retain the equally weighted average of the source updates and the rest are set to zero; the mask uses only source-side information and does no norm compensation.
    • Cos-Select+SN: retains this mask and scales the selected updates to the same global norm as SN-TRV, for comparing different update constructions at equal norm; WSC [27] serves as another calibration baseline.
  4. What experiments did the paper conduct?

    Fixed/adaptive EDPA transfer, defense recipes, and magnitude ablations are performed on three LIBERO backbones and validated on the PiPER real robot; SN-TRV outperforms Mean-∆ in most settings.

    Experimental setup

    • Policies under test: OpenVLA-OFT (main backbone, main camera + wrist camera + proprioceptive input), OpenVLA, π0; within each backbone the source and target encoder architectures are aligned.
    • Benchmark: The four LIBERO suites Spatial, Object, Goal, Long, with 10 tasks per suite; the default sources are Spatial and Object, and Goal and Long are held-out targets.
    • Sample size: Unless otherwise stated, 50 rollouts per task, i.e., 500 episodes per suite-level condition; 100 trials per task–condition pair on the real robot.
    • Attacks: EDPA (Primary/Wrist/Dual), with patches by default optimized against the original target policy; the adaptive setting re-optimizes against each victim policy with the same 50k-step budget; AGSD attacks are also used.
    • Baselines and references: Original, Spatial ∆, Object ∆, Mean-∆, Cos-Select, WSC [27]; EDPA-AF is a robustness reference that uses target adversarial data and target-side optimization, and the authors explicitly state it is not a budget-matched transfer baseline.
    • Metrics: Task success rate SR (%); Table II additionally reports the Fixed−Adaptive Drop (percentage points).

    Main results

    Table I: Cross-task transfer on OpenVLA-OFT, EDPA attacks, values are success rates (%).

    Swipe horizontally to read the table

    TargetMethodCleanPrimaryWristDual
    GoalOriginal97.288.861.422.2
    GoalSpatial ∆93.090.092.487.2
    GoalMean-∆94.684.481.476.6
    GoalEDPA-AF [5]96.497.096.296.2
    GoalSN-TRV97.297.097.295.4
    LongOriginal93.268.448.210.0
    LongSpatial ∆90.279.881.668.6
    LongMean-∆90.476.876.462.0
    LongEDPA-AF [5]92.686.890.885.2
    LongSN-TRV92.486.287.679.4

    (According to Table I; the Object ∆ and Cos-Select rows are omitted, with Dual values of 80.0/80.8 on Goal and 60.6/58.2 on Long.)

    • Single-source transfer is already effective: The authors note that directly transferring the Spatial update raises the Dual success rate from 22.2%/10.0% to 87.2%/68.6% (increases of 65.0 and 58.6 percentage points), indicating that cross-task defense reuse does not depend on multi-source aggregation.
    • Naive averaging is instead worse: Mean-∆'s Dual success rates (76.6%/62.0%) are lower than direct Spatial transfer; SN-TRV is 18.8 and 17.4 percentage points higher than Mean-∆, with clean success rates maintained at 97.2%/92.4%.
    • Gap to target-side fine-tuning: Averaged over Goal and Long, SN-TRV's Dual success rate is 87.4%, versus 90.7% for the target-specific EDPA-AF (the latter requires 50k steps of optimization per target, and this comparison assumes the source robust policy is already available and does not count its training cost).

    Adaptive attacks

    • What was tested: Table II re-optimizes Dual EDPA against each victim policy with a 50k-step budget, to test whether the fixed-patch gains come from attack–model mismatch.
    • Results: Under adaptive Dual, SN-TRV rises from Mean-∆'s 61.4% to 81.2% on Goal, and from 28.6% to 46.6% on Long; SN-TRV's Fixed→Adaptive Drop is 14.2/32.8 percentage points, close to Mean-∆'s (15.2/33.4). On Long, SN-TRV (46.6%) is higher than EDPA-AF (43.8%), which the authors state does not constitute a general advantage over target-specific adversarial fine-tuning.
    • Authors' interpretation: The benefit of source-norm calibration persists under the adaptive protocol; the main advantage is higher absolute success rate, not a smaller fixed-to-adaptive drop.

    Defense recipe and attack coverage

    • What was tested: Table III compares two source updates, EDPA-AF and SARF, on OpenVLA, evaluated against EDPA and AGSD attacks respectively.
    • Results: SN-TRV with the EDPA-AF source achieves 59.2%/27.4% (Goal/Long) against EDPA, but only 2.0%/0.8% against AGSD; SN-TRV with the SARF source achieves 72.4%/44.0% against EDPA and 71.8%/43.2% against AGSD. The gaps to the corresponding defense references are 0.0–2.2 percentage points, which the authors describe as closeness in point estimates, not statistical equivalence.
    • Authors' interpretation: The source defense recipe determines the attack coverage after transfer, and SN-TRV handles aggregation and magnitude calibration; calibrating EDPA-AF updates does not make them effective against AGSD.

    Cross-backbone generalization and real-robot experiments

    • Cross-backbone (Table IV): On OpenVLA, SN-TRV raises the EDPA success rate from 0.0% to 59.2% (Goal) and 27.4% (Long); on π0, from 49.6% to 89.8% (Goal) and from 32.8% to 71.4% (Long), but π0 Long's clean success rate drops from 81.0% to 75.8%. The authors state that the clean-performance cost varies with architecture and task.
    • Real robot (Table VI): On PiPER with four tasks and printed EDPA patches, the original policy's average success rate drops from 68.0% clean to 20.3%; SN-TRV rises to 45.8%, higher than Single-∆'s 27.5% and lower than the target-specific EDPA-AF's 48.5%; the advantage over Single-∆ holds on all four tasks (e.g., Long Task 40.0% vs. 22.0%).

    Other ablations and analyses

    • Direction–magnitude analysis (Fig. 4): The global cosine similarity of the Spatial and Object updates is 0.0942; the mean norm 7.841 is below the mean source norm 10.600; SN-TRV restores the norm to 10.600 in the same direction (about ×1.352).
    • Magnitude sweep (Fig. 5): λ ∈ {1.0, 1.25, λSN, √2, 2.0}; at λSN ≈ 1.352 the clean and fixed Dual success rates are the highest among the five values; at λ = 2, Dual drops to 74.2%/54.4% and clean drops to 86.4%/76.8%. The authors state this supports a source-derived magnitude reference, not global optimality or a statistical advantage.
    • Update construction (Table V): Cos-Select's original norm is 5.477; at equal norm (10.600), Cos-Select+SN's Dual is 89.4%/70.6%, still below SN-TRV's 95.4%/79.4%; WSC is 84.2%/68.8%.
    • Single-camera vs. dual-camera (Table I): On Long, Original is 68.4%/48.2% under Primary/Wrist, and only 10.0% under Dual; the authors use this to point out that good performance under single-view attacks does not represent performance when both views are perturbed simultaneously.
  5. What are some points that can be further explored?

    The authors note that transfer is limited by parameter-space compatibility and the attack coverage of the source defense, and that the clean-performance cost varies with task and architecture.

    Limitations and future directions noted by the authors

    • Parameter-space compatibility (Conclusion): The authors state that the method's applicability depends on a compatible parameter space; the experiments only perform coordinate-aligned transfer within the same backbone, not across incompatible architectures.
    • Attack coverage (Conclusion and Section IV-E): The transferred defense retains the attack coverage pattern learned by the source defense rather than producing a new attack-agnostic defense; updates from the EDPA-AF source achieve only 2.0%/0.8% against AGSD (Table III).
    • Clean-performance cost (Conclusion and Section IV-F): The loss in clean performance varies with task and architecture, e.g., π0 Long's clean success rate drops from 81.0% to 75.8% (Table IV).
    • Scope of magnitude calibration (Section IV-G): The authors state that the magnitude sweep supports a "source-derived magnitude reference" but does not constitute a global optimum on a continuous scale, nor a statistical advantage over neighboring values; SN-TRV's geometric characterization does not guarantee optimal target robustness (Section III-C).
    • Relationship to target-side fine-tuning (Section IV-D): On Long, SN-TRV's adaptive success rate being higher than EDPA-AF's (46.6% vs. 43.8%) does not establish a general advantage over target-specific adversarial fine-tuning; the gaps to the defense references are point estimates, not statistical equivalence (Section IV-E).

    Scope of experiments

    • The backbones under test are OpenVLA-OFT, OpenVLA, and π0, 3 in total; the simulation benchmark is the four LIBERO suites Spatial, Object, Goal, Long (10 tasks per suite, 500 episodes per condition).
    • Attacks cover EDPA (Primary/Wrist/Dual, fixed and 50k-step adaptive re-optimization) and AGSD; defense recipes cover two sources, EDPA-AF and SARF.
    • Real-robot evaluation is on 4 tasks on PiPER (Pick & Place, Open Drawer, Stack Cube, Long Task), with 100 trials per task–condition, using printed EDPA patches.
    • The source suites are fixed as Spatial and Object, and the targets as Goal and Long; the paper does not report experiments with other source/target suite combinations or with more sources.
    • The paper does not report confidence intervals or statistical significance tests for the success-rate measurements; the adaptive evaluation uses the same 50k-step attack budget as the fixed attacks (Section IV-D).
  6. Summarize the main content of the paper

    The authors propose and validate cross-task adversarial defense reuse: visual defense updates from a source task can be transferred to an independently trained VLA target policy, with SN-TRV handling multi-source aggregation.

    A paper on adversarial defense for VLA policies; the core question is: can robustness learned on one manipulation task be reused on another independently specialized policy without target-side adversarial fine-tuning.

    • Method: The parameter difference of the vision encoder before and after EDPA adversarial fine-tuning on the source task is used as the defense vector; SN-TRV keeps the direction of the multi-source update mean, calibrates the magnitude with the mean of the source update norms, and adds it to the target vision encoder, requiring no target adversarial data, no target-side gradient optimization, and no additional inference module.
    • Single-source transfer already works: On OpenVLA-OFT, directly transferring the Spatial update raises the fixed Dual EDPA success rate from 22.2%/10.0% (Goal/Long) to 87.2%/68.6% (Table I).
    • Multi-source aggregation needs calibration: Naive Mean-∆ suffers norm decay due to weak alignment among source updates (cosine similarity 0.0942), with Dual only 76.6%/62.0%; SN-TRV reaches 95.4%/79.4%, with clean success rates maintained at 97.2%/92.4% (Table I, Fig. 4).
    • Still effective under adaptive attacks: Under Dual EDPA re-optimized for 50k steps, SN-TRV is 81.2%/46.6% on Goal/Long, higher than Mean-∆'s 61.4%/28.6% (Table II).
    • Generalization and cost: Transfer gains are also observed on OpenVLA and π0 (π0 Goal rises from 49.6% to 89.8%), but π0 Long clean success rate drops from 81.0% to 75.8% (Table IV); under the PiPER real-robot printed patch, the average success rate rises from 20.3% to 45.8%, still below the target-specific EDPA-AF's 48.5% (Table VI).
    • Authors' conclusion: Cross-task defense reuse yields gains on three backbones, under fixed and adaptive attacks, and on the real robot; but the authors emphasize that its applicability depends on a compatible parameter space and on the attack coverage of the source defense (EDPA-AF source updates are nearly ineffective against AGSD), the clean-performance cost varies with task and architecture, and the magnitude calibration is a source-side reference rather than an optimality guarantee.
Read originalarxiv.org