LiDAR-SHIELD: A fully Dockerized adversarial robustness evaluation framework for LiDAR 3D perception models
LiDAR-SHIELD: A Fully Dockerized Adversarial Robustness Assessment Framework for LiDAR-Based 3D Perception Models
The authors propose LiDAR-SHIELD, a Dockerized framework for evaluating the adversarial robustness of LiDAR 3D perception, which connects to arbitrary models via three HTTP endpoints and supports white-box/black-box attack evaluation for both segmentation and detection tasks.
- LiDAR 3D perception models need adversarial stress testing before deployment, but existing attack implementations are scattered across their own codebases and tied to specific architectures, general-purpose adversarial toolboxes do not support point cloud inputs, and there is a lack of a unified, reproducible evaluation tool covering both segmentation and detection.
- LiDAR-SHIELD adopts a service-oriented design: the target model stays in its native environment and communicates with the evaluation engine through three HTTP endpoints, /info, /forward, and /gradient; onboarding a new model or dataset requires only a loader or adapter. The framework integrates attacks per task: PGD and Square Attack for segmentation, and point perturbation, point deletion, point injection, and LiDAttack for detection.
- SECOND, PointPillars, LENet, CENet, and FIDNet were evaluated on KITTI, CARLA, SemanticKITTI, and SemanticPOSS. Point perturbation affects PointPillars most (mAP ratio on KITTI drops to 10.59%); the black-box Square Attack reduces LENet's mIoU on SemanticKITTI from 64.70 to 19.50; LiDAttack reduces SECOND's mAP on KITTI from 70.43 to 52.51.
- The paper does not specifically discuss limitations. Experimental coverage: detection tested only SECOND and PointPillars, segmentation backbone comparison covered only LENet, CENet, FIDNet; each black-box attack was evaluated on only one model-dataset combination; the paper does not report sample sizes, number of repetitions, or statistical tests.
- Threat model
- The framework supports two threat models: white-box attacks obtain input gradients via the /gradient endpoint (e.g., PGD, point perturbation/deletion/injection), and black-box attacks access model outputs only via the /forward endpoint (e.g., Square Attack, LiDAttack); the victim systems are LiDAR 3D semantic segmentation and 3D object detection models.
- Evaluated models
- SECONDPointPillarsLENetCENetFIDNet
- Benchmarks
- KITTICARLASemanticKITTISemanticPOSS
- Metrics
- mAPmAP ratioChamfer DistancemIoU
The researchers release LiDAR-SHIELD, a fully Dockerized adversarial robustness evaluation framework for LiDAR-based 3D perception models, with code open-sourced at https://github.com/alexandrosgk/LiDAR-shield。针对现有对抗攻击实现分散在各仓库、绑定特定架构且需手动重建环境的问题,该框架不把目标模型装进工具包,而是通过; three lightweight HTTP endpoints /info, /forward and /gradient expose any model, and integrating a new model or dataset requires only a loader function or adapter. The framework supports both 3D semantic segmentation and 3D object detection through the same engine service protocol, and integrates a task-aware attack suite: for segmentation, PGD and Square Attack; for detection, point perturbation, point deletion, point injection and LiDAttack. It can systematically expose edge cases and vulnerabilities in heterogeneous LiDAR backbones before real deployment, and is applicable to autonomous driving and robotics scenarios.
Paper analysis
What problem does this paper attempt to solve?
The authors state that existing LiDAR adversarial attack implementations are scattered and tied to specific architectures, and that there is a lack of a unified, reproducible evaluation tool covering both segmentation and detection.
Core problem: how to systematically evaluate the adversarial robustness of LiDAR 3D perception models (segmentation and detection) without manual environment reconstruction and model adaptation.
- Scenario and motivation: LiDAR is the primary sensing modality for autonomous driving and robotics, and 3D object detection and 3D semantic segmentation are two core tasks; the authors argue that models must be stress-tested against adversarial inputs before deployment to expose corner cases and vulnerabilities that clean benchmarks cannot expose.
- Shortcomings of existing methods: the authors state that general adversarial toolboxes (ART, Foolbox, CleverHans, secml) adopt a library-centric integration model, requiring the target model to be installed into the toolbox codebase, and do not support LiDAR point cloud inputs; LiDAR-level work either covers only a single task, or targets natural corruption (Robo3D), or has attack code coupled to a specific architecture, so reuse requires manually rebuilding the environment and writing glue code.
- Core observation: detection and segmentation share the same upstream LiDAR measurement, and perturbations in the input space can endanger both tasks simultaneously, so a single engine-service protocol can uniformly evaluate them.
- Proposed approach: LiDAR-SHIELD, a unified, fully Dockerized adversarial robustness evaluation framework that connects to any model through three HTTP endpoints (/info, /forward, /gradient) and integrates a task-aware attack suite (segmentation: PGD, Square Attack; detection: point perturbation, point deletion, point injection, LiDAttack).
- Threat model (attack settings supported by the framework):
- Attacker goal: reduce segmentation mIoU or detection mAP, i.e., maximize the task loss.
- Knowledge: two levels, white-box (obtaining input gradients via /gradient) and black-box (obtaining only prediction outputs via /forward).
- Capability: add ℓp-bounded perturbations to the range-view tensor (segmentation), or perform coordinate perturbation, deletion, and injection on the point cloud (detection), constrained by an ϵ budget or Chamfer distance.
- Victim systems: LiDAR 3D semantic segmentation models and 3D object detection models.
What related research exists?
Related work falls into three groups: LiDAR adversarial attacks, LiDAR perception model families, and robustness toolboxes and benchmarks; the authors state that no existing tool uniformly covers both task types.
Adversarial attacks on LiDAR perception
- White-box digital perturbations can suppress 3D detection or introduce localization errors [19], and adversarial examples can transfer across LiDAR architectures [18]; the vulnerability of segmentation pipelines under digital perturbations has also been studied [20], [33].
- At the point cloud level: deleting points alone can fool 3D models [21], and transferable perturbations on raw point clouds [22]; Zhang et al. [31] systematically studied the adversarial robustness of LiDAR 3D detectors, and Chen et al. [29] proposed the black-box attack LiDAttack.
- The authors note that these attack implementations do not provide reusable evaluation infrastructure covering both detection and segmentation.
LiDAR perception models
- Segmentation: RangeNet++ [8], CENet [9], FIDNet [10], and LENet [11] represent different design points for dense scene understanding.
- Detection: PointPillars [3], VoxelNet [5], SECOND [4], and CenterPoint [6] represent pillar-based, voxel-based, and center-based pipelines; the authors argue that this heterogeneity requires a service abstraction that can accommodate arbitrary backbones.
Robustness toolboxes and benchmarks
- General-purpose toolboxes ART [25], Foolbox [26], CleverHans [27], and secml [35] require embedding the target model into their codebases; the authors note that they do not provide native support for LiDAR point cloud inputs or 3D perception outputs.
- Ji et al. [32] benchmarked adversarial robustness for point cloud recognition, but for classification rather than detection and segmentation; the benchmark of Zhang et al. [31] covers only detection; Robo3D [30] covers detection and segmentation but targets natural corruption rather than adversarial attacks.
Baselines and benchmarks: This paper does not propose new attacks, but integrates existing attacks as a built-in suite—PGD [13] (white-box) and Square Attack [17] (black-box) for segmentation, and point perturbation/deletion/injection (white-box, following [19], [21], [22]) and LiDAttack [29] (black-box) for detection; the evaluation datasets are KITTI, CARLA, SemanticKITTI, and SemanticPOSS.
The authors position this paper as filling the gap of "no unified, Dockerized adversarial evaluation tool covering both LiDAR segmentation and detection."
How does the paper address this problem?
Service-oriented architecture: the model stays in its native environment and communicates with the evaluation engine through three HTTP endpoints, and attacks are instantiated per task.
The overall idea is to decompose robustness evaluation into three Dockerized components, using two contracts (data contract, model contract) to isolate dataset differences and model differences at the system boundary, so that the LiDAR-SHIELD evaluation engine remains task-aware but model-agnostic.
Architecture and components
- User Client / Data Adapter: responsible for loading LiDAR samples and annotations and packaging them into task-tagged requests; all benchmark-related logic (file layout, calibration, annotation parsing) stays in this layer, and onboarding a new dataset requires only implementing this adapter.
- Assessment Engine: the core of the method, responsible for selecting attacks, maintaining perturbation state, coordinating clean/adversarial forward passes, and executing threat-model control logic; attack behavior is implemented only once and reused across models.
- Model Service: encapsulates the target network's construction, checkpoint loading, task loss definition, and pre/post-processing; onboarding a new backbone requires only implementing load_model(device), returning (model, criterion, info, device).
Interface contracts
- data contract: specifies the format of the sample payload output by the adapter; the engine does not directly parse raw dataset files.
- model contract: three endpoints—/info returns model and task metadata, /forward returns task-native predictions, and /gradient returns the input gradients required for white-box attacks; the paper states that any backbone implementing these three calls can be attacked without importing its codebase.
Engine workflow
- Each model session queries /info once to obtain metadata and instantiate the task processing path.
- For a sample (x, y), a clean reference prediction is obtained via /forward, the attack is instantiated according to the selected threat model, and an adversarial input xadv is generated iteratively.
- Adversarial samples are executed through the same service boundary, and task-native outputs are collected for evaluation; black-box mode uses only /forward, white-box mode additionally calls /gradient, and the orchestration layer is unchanged.
Segmentation attack suite
- Perturb the projected range-view tensor Y: Yadv = Y + Δ, |Δ|p ≤ ϵ, with the goal of maximizing the segmentation loss.
- PGD (white-box): iterates via /gradient, p ∈ {2, ∞}, including projection and norm-specific gradient normalization.
- Square Attack (black-box): uses only /forward, performing query-efficient random search over random square regions of the input tensor under the ℓ∞ constraint.
Detection attack suite
- Point perturbation (white-box): Padv = {pi + δi}, constraining each point's displacement ∥δ_i∥2 ≤ ϵ, using PGD to maximize the detection loss.
- Point deletion (white-box): scores points by the ℓ2 norm of the detection loss gradient and greedily deletes them in descending order of saliency.
- Point injection (white-box): injects a point set Z, Padv = P ∪ Z, initializing at the highest-saliency locations and optimizing only Z under a Chamfer distance constraint.
- LiDAttack (black-box): uses only /forward, searching for point cloud modifications with a genetic algorithm plus simulated annealing.
Key parameters
- Detection white-box: point perturbation ϵ ∈ {5, 7, 10} cm, 40 iterations, step size α = ϵ/30; point deletion 10 iterations, deletion rate {0.5, 1.0, 1.5}%; point injection 300 points, ϵ ∈ {0.3, 0.5, 0.7} m.
- Detection black-box: LiDAttack optimizes K = 20 points, within a 0.5 m margin of the target box, with at most 1000 forward queries per target.
- Segmentation white-box: PGD ℓ2 ϵ ∈ {3, 6, 9}, ℓ∞ ϵ ∈ {0.1, 0.3, 0.5}, T = 40, α = ϵ/T; black-box Square Attack: ℓ∞ ϵ = 0.3, 1000 queries, one random restart, initial perturbation probability pinit = 0.4.
What experiments did the paper conduct?
The built-in attacks were evaluated on 4 datasets and 5 models: point perturbation has the greatest impact on PointPillars, and black-box attacks also degrade segmentation and detection performance.
Experimental setup
- Models tested: detection is SECOND (voxel-based) and PointPillars (pillar-based); segmentation is three pretrained range-view models, LENet, CENet, and FIDNet.
- Datasets: detection uses KITTI (64-beam real LiDAR, results at moderate difficulty) and CARLA simulated data (Town06, 3,000 frames, 10 Hz, 64-beam); segmentation uses SemanticKITTI (range-view 64×1024) and SemanticPOSS (40×1800).
- Attack configuration: see the parameters in question 3; each black-box attack is evaluated on only one combination—LiDAttack attacks SECOND on KITTI, and Square Attack attacks LENet on SemanticKITTI.
- Metrics: detection uses absolute mAP, mAP ratio (mAPadv/mAPclean × 100, lower means stronger degradation), and symmetric Chamfer Distance (CD, lower means better geometric fidelity); segmentation uses mIoU.
- Sample sizes and repetitions: the paper does not report the number of samples, number of repetitions, or statistical tests for each experiment.
Main results
Table II gives the mAP ratio (%) and CD for detection; the table below lists only the mAP ratio:
Swipe horizontally to read the table
Dataset Detector Point perturbation ϵ=5cm ϵ=7cm ϵ=10cm Point deletion 0.5% 1.0% 1.5% Point injection ϵ=0.3m 0.5m 0.7m KITTI SECOND 90.91 83.10 65.48 83.15 74.18 67.98 71.73 65.28 71.12 KITTI PointPillars 33.02 19.56 10.59 80.25 62.07 57.40 87.85 75.83 76.76 CARLA SECOND 89.06 82.80 73.21 95.28 91.77 85.32 87.72 78.46 77.24 CARLA PointPillars 22.82 13.87 3.46 89.97 85.23 80.02 95.93 93.90 89.72 (According to Table II; CD increases as the perturbation budget grows, omitted here)
- The authors state that point perturbation causes the strongest degradation, with the mAP ratio continuously decreasing as the displacement budget grows, and the effect on PointPillars is especially large; the authors believe that voxel-based sparse convolution detectors are less sensitive to coordinate-level perturbations than pillar-based encoders.
- The authors state that point deletion causes gentler degradation; point injection causes moderate degradation with lower CD.
- Exception: on KITTI, increasing the injection radius does not always strengthen the attack—SECOND's mAP ratio drops to 65.28% at ϵ=0.5 m and rises back to 71.12% at ϵ=0.7 m, and PointPillars is similar; the authors believe that an excessively large budget moves the injected points out of the most influential regions.
Segmentation robustness (Table III, Table V)
- LENet on SemanticKITTI: mIoU drops from 64.7 to 41.7 at ℓ2 ϵ=9 and to 34.7 at ℓ∞ ϵ=0.5; on SemanticPOSS it drops from 52.7 to 34.4 for ℓ2 and 28.8 for ℓ∞ (Table III).
- Backbone comparison (SemanticKITTI, ℓ2 ϵ=6, Table V): LENet 64.70→46.90, CENet 61.70→44.30, FIDNet 60.80→44.10; the authors state that all three degrade under attack.
Black-box attacks (Table IV)
- Square Attack segmentation: LENet's mIoU on SemanticKITTI drops from 64.70 to 19.50; the authors state that this confirms vulnerability without gradient access.
- LiDAttack detection: SECOND's mAP on KITTI drops from 70.43 to 52.51, which the authors state is a relative decrease of about 25%.
Other ablations and analyses
- The paper does not report ablations other than the perturbation budget sweeps (the ϵ and deletion rate values in Table II and Table III).
What are some points that can be further explored?
The paper does not specifically discuss limitations; the experiments cover 5 models and 4 datasets, each black-box attack is tested on only one combination, and sample sizes and number of repetitions are not reported.
Limitations and future directions noted by the authors
- The paper does not specifically discuss limitations, and there is no separate Limitations or Future Work section; the Conclusion only summarizes the framework's capabilities and experimental conclusions.
Scope of experimental coverage
- Models tested: detection is SECOND and PointPillars, 2 in total; segmentation backbone comparison (Table V) is LENet, CENet, FIDNet, 3 in total; the segmentation budget sweep (Table III) and the black-box Square Attack are conducted only on LENet.
- Datasets: detection uses KITTI (moderate difficulty) and CARLA (Town06, 3,000 frames); segmentation uses SemanticKITTI and SemanticPOSS.
- Black-box attack coverage: LiDAttack attacks SECOND only on KITTI, Square Attack attacks LENet only on SemanticKITTI (Table IV), each being one model-dataset combination.
- Attack coverage: the built-in suite is PGD and Square Attack for segmentation and point perturbation, point deletion, point injection, and LiDAttack for detection (Table I); the introduction mentions that sensor-level physical attacks exist, but the experiments include only digital perturbations.
- Not reported by the paper: sample sizes, number of repetitions, statistical significance tests for each experiment, and the actual time cost or query overhead of the attacks (only the budget caps of at most 1000 queries per target for LiDAttack and 1000 queries for Square Attack are given).
Summarize the main content of the paper
LiDAR-SHIELD uses three HTTP endpoints to decouple adversarial evaluation from the target model, with a single engine covering white-box/black-box attack evaluation for segmentation and detection.
LiDAR-SHIELD is a fully Dockerized adversarial robustness evaluation framework for LiDAR 3D perception, targeting pre-deployment stress testing for autonomous driving and robotics.
- Problem: the authors state that existing attack implementations are scattered and tied to specific architectures and dependency stacks, general toolboxes do not support point cloud inputs, and there is a lack of a unified, reproducible tool covering both 3D semantic segmentation and 3D object detection.
- Method: service-oriented design—the target model stays in its native environment and communicates with the evaluation engine through three HTTP endpoints, /info, /forward, and /gradient; onboarding a new model or dataset requires only a loader or adapter. Attacks are instantiated per task: segmentation performs PGD (white-box) and Square Attack (black-box) on the range-view tensor, and detection performs point perturbation, point deletion, and point injection (white-box) and LiDAttack (black-box) on the point cloud.
- Main results:
- Point perturbation affects PointPillars most: on KITTI the mAP ratio drops to 10.59% at ϵ=10 cm, and to 3.46% on CARLA; SECOND retains 65.48% and 73.21% over the same period (Table II).
- Black-box attacks are effective: Square Attack reduces LENet's mIoU on SemanticKITTI from 64.70 to 19.50; LiDAttack reduces SECOND's mAP on KITTI from 70.43 to 52.51 (Table IV).
- Segmentation backbone comparison (SemanticKITTI, ℓ2 ϵ=6): the mIoU of LENet, CENet, and FIDNet drops from 64.70, 61.70, and 60.80 to 46.90, 44.30, and 44.10, respectively (Table V).
- Exception: on KITTI, when the point injection budget increases from 0.5 m to 0.7 m, the attack instead weakens (SECOND mAP ratio 65.28%→71.12%).
- Authors' conclusion: the authors state that the experiments on SemanticKITTI, SemanticPOSS, KITTI, and CARLA confirm that the integrated attacks can expose vulnerabilities of heterogeneous LiDAR backbones; the code is publicly available.