Skip to content

VLAN: Vision-Language Accessible Navigation

VLAN: Vision-Language Accessible Navigation

arXiv· arXiv:2611.18346· Jie Hu, Yu Zheng, Wenzhi Wu, Weixing Feng, Cunlei Cheng, Jiaming ZhangHunan UniversityIndexed by this site · 4 days ago· Originally publishedEvaluation· Its place in the scan record →

Quick read

Evaluation and benchmarks

Based on the paper PDF (AI-generated); refer to the original

The authors propose a multi-user accessibility navigation benchmark, VLAN (1000 scenarios, 20K trajectories), and evaluate 7 open-source VLMs; the authors state that the models struggle to adjust safety standards across five user types.

Questions
Existing navigation benchmarks assume a fixed embodiment and user-independent passability criteria, and cannot evaluate the different accessibility of the same obstacle for different users (robots, BVI users, wheelchair users).
Methods
The authors reconstruct first-person videos into editable Gaussian/Unity scenes, generating 1000 scenes, 20K trajectories, and 160K keyframes, label Passable/Bypass/HighRisk/Blocked by five user types, and design two task types—action prediction and language guidance—along with metrics.
Experiments and results
7 open-source VLMs completed all five user settings: InternVL3.5-8B has the highest overall AccSR (0.426); the authors state that models struggle to consistently adjust obstacle relevance, path selection and safety standards across users; the ranking is largely maintained on 30 real trajectories.
Limitations and future work
The authors note: users with hearing or cognitive impairments are not covered; the four-category labels cannot represent graded risk; the real-world evaluation has only 30 trajectories; results are from a single greedy decoding run with no error bars; the Drone task is limited to fixed-height 2D.
Experimental setupQwen2.5-VL-3B-Instruct、Qwen2.5-VL-7B-Instruct et al. · VLAN · SRact、GR et al.
Evaluated models
Qwen2.5-VL-3B-InstructQwen2.5-VL-7B-InstructQwen3-VL-8B-InstructInternVL3.5-8BIdefics3-8B-Llama3Molmo2-8BLlama-3.2-11B-Vision-InstructGPT-4oGemini 3.6
Benchmarks
VLAN
Metrics
SRactGRCFSPLSRlanIASAccSR
AI summaryJie Hu and other researchers propose VLAN, a vision-language accessibility navigation benchmark containing 1,000 scenes, 20K navigation trajectories and 160K keyframes, covering diverse lighting, environmental conditions, scene semantics and path geometries. The benchmark provides embodiment-specific actions and accessibility features for legged and wheeled agents, covering humanoid and quadruped robots as well as people with visual or mobility impairments, with everyday obstacles and accessibility labels divided into four categories: Passable, Bypass, HighRisk and Blocked. Unlike evaluations that only assess whether the goal can be reached, the benchmark measures physical safety and adherence to user-specific constraints through goal achievement, collision awareness and accessibility alignment metrics. Evaluations of modern vision-language models show that they often fail to consistently adjust obstacle relevance, path selection and safety criteria according to user profiles, based on which the authors argue that accessibility-aware navigation requires dedicated datasets, tasks and metrics beyond conventional embodied navigation evaluation. Data and code are open-sourced at https://github.com/InSAI-Lab/VLAN。

Jie Hu and other researchers propose VLAN, a vision-language accessibility navigation benchmark containing 1,000 scenes, 20K navigation trajectories and 160K keyframes, covering diverse lighting, environmental conditions, scene semantics and path geometries. The benchmark provides embodiment-specific actions and accessibility features for legged and wheeled agents, covering humanoid and quadruped robots as well as people with visual or mobility impairments, with everyday obstacles and accessibility labels divided into four categories: Passable, Bypass, HighRisk and Blocked. Unlike evaluations that only assess whether the goal can be reached, the benchmark measures physical safety and adherence to user-specific constraints through goal achievement, collision awareness and accessibility alignment metrics. Evaluations of modern vision-language models show that they often fail to consistently adjust obstacle relevance, path selection and safety criteria according to user profiles, based on which the authors argue that accessibility-aware navigation requires dedicated datasets, tasks and metrics beyond conventional embodied navigation evaluation. Data and code are open-sourced at https://github.com/InSAI-Lab/VLAN。

Paper analysis

6 questions; open one to read the full answer

  1. What problem does this paper attempt to solve?

    The authors propose VLAN, a multi-user accessibility navigation benchmark, to evaluate whether VLMs can adjust navigation behavior and safety standards according to user type.

    Core problem: existing navigation benchmarks assume fixed morphology and user-independent traversability standards, and cannot evaluate cases where the same visual scene has different reachability for different users (robots, BVI users, wheelchair users).

    • Scenario and importance: the authors note that reachability is user-conditioned rather than inherent to the scene—the same low obstacle is traversable for a humanoid robot, high-risk for a BVI user, and blocking for a wheelchair user; reaching the goal but with an unsafe trajectory or one that violates user constraints does not count as success.
    • Shortcomings of existing methods: the authors state that existing benchmarks (e.g., R2R, VLN-CE) typically assume fixed morphology and use "reaching the goal with no collision for a generic agent" as the success criterion, lacking user-specific action and reachability annotations for multiple human and robot users in the same scene.
    • Core observation: editable reconstructed scenes preserve realistic visual appearance while supporting interactive scene modification, trajectory generation, keyframe rendering and metric-based evaluation, thereby scaling data without additional real-world collection.
    • Paper proposes: the VLAN benchmark, with 1000 scenes, 20K trajectories, 160K keyframes, covering five user morphologies (humanoid, quadruped, drone, BVI, wheelchair), providing user-conditioned actions, four reachability labels (Passable/Bypass/HighRisk/Blocked), and collision-aware, reachability-aligned evaluation metrics; the authors state this is the first time five morphologies and accessibility features are incorporated into navigation evaluation.
  2. What related research exists?

    Related work falls into three groups: embodied and capability-conditioned navigation, assistive and accessibility benchmarks, and real-to-interactive simulation; the authors state that VLAN is distinct from them.

    Embodied and capability-conditioned navigation

    • R2R, RxR, VLN-CE—advance instruction-following navigation from discrete graphs to continuous embodied environments; the authors note that they mostly assume fixed morphology and user-independent traversability.
    • CapNav, NaviTrace, VLN-PE—evaluate capability-conditioned indoor navigation, morphology-conditioned image-space trajectories, and gaps in cross-morphology VLN, respectively.
    • EmbodiedCity, Embodied4C—introduce morphology switching among vehicles, drones, robotic arms, etc.; the authors note that they are robot/vehicle-centric, have no human users with perceptual or mobility impairments, no user-conditioned reachability labels, and do not test whether a model gives different outputs to different users under the same visual input.

    Assistive and accessibility benchmarks

    • VIALM—the authors note that large models may lack environment grounding and fine-grained guidance for visually impaired users.
    • @BENCH—evaluates VLM performance on assistive tasks chosen by visually impaired people (segmentation, depth estimation, OCR, captioning, VQA).
    • WalkVLM—builds a video benchmark for visually impaired walking assistance. The authors note that these works lean toward perception, description or reminders, whereas VLAN provides action-level and instruction-level navigation evaluation.

    Real-to-interactive simulation

    • Vid2Sim—converts monocular video into a photorealistic interactive urban navigation simulation environment; RoomTour3D—builds geometry-aware data from room tour videos.
    • EmbodiedSplat, GaussGym, RoboGSim—explore Gaussian splatting-based interactive environments. The authors state that VLAN follows this direction but is used to build a multi-user accessibility navigation benchmark.

    Baselines and benchmark: the baselines compared in the paper are 7 open-source MLLMs (Qwen2.5-VL-3B/7B, Qwen3-VL-8B, InternVL3.5-8B, Idefics3-8B, Molmo2-8B, Llama-3.2-11B-Vision), plus GPT-4o and Gemini 3.6 evaluated on a 200-item subset as commercial references; the evaluation benchmark is the paper's own VLAN. Authors' positioning: VLAN evaluates multiple human and robot users in the same scene with user-specific actions, reachability labels and safety-aware metrics, distinguishing it from all the above work.

  3. How does the paper address this problem?

    VLAN uses editable reconstructed scenes to generate user-conditioned trajectories and labels, with two task types, action prediction and language-guided, and includes collision-aware and reachability-aligned metrics.

    Overall approach: reconstruct first-person videos into editable Gaussian/Unity scenes, and under the same scene, same start and goal points and obstacle layout, generate different navigation trajectories, actions and reachability labels for five user morphologies, then evaluate general VLMs with VLAN's user-conditioned metrics.

    Data construction

    • Scene reconstruction and editing: first-person scene videos are reconstructed into Gaussian-splat representations and imported into Unity for controlled editing (traffic light states, obstacle configurations, route conditions); one base scene can produce multiple navigation instances (Figure 2, 3).
    • Scale and diversity: 1000 categorized scenes, 20K trajectories, 160K keyframes, about 20 trajectories per scene and 8 keyframes per trajectory; covering lighting (Daytime 474, Dusk 131, Nighttime 251, Low Vision 144), scene types, path geometry, weather/ground conditions (Table 1).
    • Obstacle categories: by function, divided into four categories: Navigation (barriers, cones, temporary fences), Ground (low obstacles, curbs, boxes, uneven surfaces), Overhead (branches, hanging signs, canopies), Traffic-sign.

    User-conditioned trajectories and actions

    • Reachability labels: manually annotated; each route is labeled for each user as one of Passable, Bypass, HighRisk, Blocked; labels are user-conditioned, e.g., a low obstacle can be bypassed by humanoid/quadruped, is high-risk for BVI, and blocks wheelchairs.
    • Trajectory generation: user-conditioned routes are defined as τu = 𝒢(𝒮, o, xs, xg, u), i.e., Unity generates trajectories under user-specific reachability constraints; the target behavior for Blocked routes is to stop or reroute rather than force arrival.
    • Expert actions: pose sequences are converted to local actions at = (Δ xt, Δ zt, Δθt) (lateral displacement, forward displacement, yaw rotation); Drone is set to fixed-height 2D navigation and shares the same action interface with Quadruped for direct comparison (Appendix B).

    Task settings

    • Action Prediction Navigation (Drone, Quadruped): the model receives the first 3 keyframes, the first 2 actions, the goal position, the last keyframe and the user morphology, and predicts the future action sequence, evaluated by offline rollout execution.
    • Language-Guided Navigation (Humanoid, BVI, Wheelchair): the model generates user-specific natural language guidance; BVI guidance uses steps rather than meters, clock directions rather than left/right, and explicit risk warnings; wheelchair guidance emphasizes feasibility, clear width and ground stability.

    Evaluation protocol

    • Action prediction tasks use SRact (success only if the goal region is reached without collision), GR, CF, SPL; language-guided tasks use SRlan (success only if all user-specific semantic constraints are satisfied) and IAS (structured similarity to the standard goal), with different constraint sets per user (Table 2).
    • Cross-user composite metric AccSR: first average the success rates of the language category (3 users) and the action category (2 users) separately, then take their harmonic mean.
  4. What experiments did the paper conduct?

    7 open-source VLMs were evaluated on five user tasks over 100 scenes and 1931 trajectories; InternVL3.5-8B has the highest overall AccSR (0.426).

    Experimental setup

    • Evaluated models: 7 open-source MLLMs complete all five user settings—Qwen2.5-VL-3B-Instruct, Qwen2.5-VL-7B-Instruct, Qwen3-VL-8B-Instruct, InternVL3.5-8B, Idefics3-8B-Llama3, Molmo2-8B, Llama-3.2-11B-Vision-Instruct; additionally GPT-4o and Gemini 3.6 are evaluated on a 200-item subset (Table 14).
    • Dataset: the VLAN test set, 100 scenes, 1931 trajectories per user setting; five user types across two task types.
    • Metrics: the action prediction task reports SRact, GR, CF, SPL; the language guidance task reports SRlan, IAS; cross-user results report AccSR.
    • Evaluation method: action prediction is judged by offline rollout (success if the endpoint is within 0.30 m of the goal and at least 0.30 m from obstacles throughout, Appendix D.4); language guidance is judged by user-specific semantic constraints (Table 2); the paper does not state whether an LLM judge is used.
    • Number of repetitions: all reported scores come from a single deterministic greedy decoding, with no cross-run variance (Appendix A).

    Main results

    According to Tables 3, 4, 5 (full test set, 1931 trajectories per user setting):

    Swipe horizontally to read the table

    ModelDrone SRactQuadruped SRactHumanoid SRlanBVI SRlanWheelchair SRlanAccSR
    Qwen2.5-VL-3B0.1330.0530.4830.2650.3320.147
    Qwen2.5-VL-7B0.5710.3840.2750.2760.2080.331
    Qwen3-VL-8B0.6220.4740.3890.4360.0920.392
    InternVL3.5-8B0.6420.3930.3650.3750.3470.426
    Idefics3-8B0.5540.3080.4380.1180.0040.260
    Molmo2-8B0.2890.1470.5230.1310.2890.258
    Llama-3.2-11B0.2470.0480.5580.1830.0750.191
    • Authors' interpretation: InternVL3.5-8B has the highest AccSR and is the most balanced across the two task types; Qwen3-VL-8B is strong at action prediction but its Wheelchair SRlan is only 0.092, lowering its composite score.
    • Authors' interpretation: Qwen2.5-VL-3B has the highest collision-free rate (Drone CF 0.947, Quadruped CF 0.983) but the lowest success rate, showing a "safe but inactive" pattern; the authors argue that strong language capability does not automatically translate into precise action rollout quality.
    • Authors' interpretation: no model leads consistently across all user types, and the authors state that current systems struggle to consistently adjust navigation behavior and safety criteria across user profiles.

    Perception and planning failure analysis

    • What was measured: a four-quadrant split by whether the language output names the obstacle category correctly (P±) and whether the action succeeds (A±) (Table 6).
    • Results: failures for Qwen2.5-VL-3B and Molmo2-8B are mainly P−A− (63.4%, 61.4%); P+A− accounts for 23.3% for Qwen2.5-VL-3B (the highest among all models); Idefics3-8B reaches 40.4% P−A+, i.e., it still reaches the goal while naming the obstacle incorrectly.
    • Authors' interpretation: the four quadrants are operational outcome categories rather than a fully decoupled causal decomposition; P+A− is the quantitative counterpart of the "safe but inactive" behavior.

    Sim-to-real evaluation

    • What was measured: 30 additional real-world trajectories were collected, evaluating the same five user tasks (Table 7).
    • Results: scores for each model are lower than on VLAN but comparable, and the ranking is largely preserved; InternVL3.5-8B AccSR 0.302 is still highest, and Qwen2.5-VL-3B 0.123 is lowest.
    • Authors' interpretation: because there are only 30 trajectories, the authors regard this as preliminary evidence that VLAN scores correlate with real-world performance, not a conclusive real-world evaluation.

    Closed-source reference models

    • What was measured: 9 models are evaluated uniformly on a fixed 200-trajectory subset (2 per scene, seed 42) (Table 14), not directly comparable to the main table.
    • Results: Gemini 3.6 is highest on Drone (0.900), Quadruped (0.546), and BVI (0.515); the highest on Humanoid is Llama-3.2-11B (0.565); the highest on Wheelchair is GPT-4o (0.382), while Gemini 3.6 drops to 0.205.
    • Authors' interpretation: closed-source capability is not consistently stronger across all user profiles, reinforcing the observation that general capability does not automatically transfer to accessibility-conditioned navigation; the subset conclusions agree with the full set (34 of 35 cells deviate by less than 0.05, and the top-4 ranking is preserved in the top five settings).

    Other ablations and analyses

    • Lighting conditions (Table 9): AccSR decreases under low visibility and dusk; InternVL3.5-8B is highest under Daytime/Low Vision/Dusk, and Qwen3-VL-8B is slightly higher under Nighttime (0.423).
    • Path geometry (Table 10): Narrow and Constrained scenes are harder, and InternVL3.5-8B is highest under all four geometry types.
    • Step-wise action statistics (Table 11, Drone): Qwen2.5-VL-3B has an average |∆θ| of 20.31°/step and a path ratio of 0.77; Llama-3.2-11B has the lowest path ratio (0.59).
    • Drone failure decomposition (Table 12): 97.2% of Qwen2.5-VL-3B's failures are action generation failures and 0.1% are collisions; Idefics3-8B has 31.0% dual failures.
    • Cross-user consistency (Table 13): for the conditioned failure rate where BVI is wrong but Humanoid is correct, Qwen3-VL-8B is 2.2% and Idefics3-8B reaches 38.4%.
    • Fixed-height analysis (Table 8): after removing 560 constrained trajectories, Drone SRact improves by +0.034 to +0.100, while Quadruped changes only from −0.012 to +0.010, with a positive ∆∆ (+0.035 to +0.098); the authors state that this supports the constrained subset being particularly difficult for aerial embodiments, but does not prove that 2D labels agree with 3D policy labels.
  5. What are some points that can be further explored?

    The authors list limitations including morphology coverage, label granularity, real-world evaluation scale, and single decoding; experiments cover 9 models and five user tasks.

    Limitations and future directions noted by the authors

    • The considered embodiments do not cover all user groups, such as people with hearing or cognitive impairments; future work could extend to broader user groups, richer multimodal inputs, and more personalized user models (Appendix A).
    • Accessibility is compressed into four decision-oriented categories and cannot represent graded risk, personal preferences, or capability profiles beyond the five embodiments (Appendix A).
    • The Drone task is defined on a fixed-height plane, and results are limited to fixed-height 2D accessibility-conditioned navigation; labels and reference actions under 3D policies are left for future work (Appendices A, B.2).
    • Sim-to-real covers only 30 real trajectories, which the authors regard as preliminary evidence of ranking consistency rather than a conclusive real-world evaluation (Appendix A).
    • All scores come from a single deterministic greedy decoding, with no cross-run variance; only point estimates are reported, with no error bars (Appendix A).
    • Annotation did not include participants with lived experience of disability; the authors believe this could strengthen the HighRisk and Blocked boundary cases and list it as a future iteration direction; future annotation could be scaled using LLM pre-annotation plus human verification (Appendix E.1).

    Experimental coverage

    • The evaluated models are 7 open-source MLLMs (3B–11B), plus GPT-4o and Gemini 3.6 evaluated on a 200-item subset, for 9 in total (Sec. 4.1, Appendix D.6).
    • The evaluation tasks cover five user types: action prediction (Drone, Quadruped) and language guidance (Humanoid, BVI, Wheelchair), with a test set of 100 scenes and 1931 trajectories per user setting (Sec. 4.1).
    • The real-world evaluation uses 30 trajectories (Table 7); the closed-source model evaluation uses a 200-trajectory subset (Table 14).
    • The paper does not report whether the language guidance task uses an LLM judge or which judge model; it does not report agreement between human evaluation and automatic metrics.
    • The authors explain that they chose general VLMs rather than task-specific navigation agents as baselines because existing VLN agents assume a single fixed embodiment and consume an interactive first-person observation stream, whereas VLAN provides a fixed keyframe set; the authors state that embodiment-specific navigation agents are a natural next step (Sec. 1).
  6. Summarize the main content of the paper

    VLAN is an accessibility navigation benchmark for five user types; evaluation of 7 open-source VLMs shows models struggle to consistently adjust safety standards across users.

    A benchmark paper: the authors build Vision-Language Accessible Navigation (VLAN) to evaluate whether vision-language models can adjust navigation behavior to different user embodiments within the same visual scene.

    • Problem: accessibility is user-conditioned—the same obstacle may be passable for a humanoid robot, high-risk for a BVI user, and blocking for a wheelchair user; the authors state that existing benchmarks assume a fixed embodiment and user-independent passability criteria, and cannot test this.
    • Method: first-person videos are reconstructed into editable Gaussian/Unity scenes, generating 1000 scenes, 20K trajectories, and 160K keyframes; five user types (humanoid, quadruped, drone, BVI, wheelchair) each have conditioned actions and Passable/Bypass/HighRisk/Blocked labels; tasks are split into action prediction (Drone, Quadruped) and language guidance (Humanoid, BVI, Wheelchair), with SRact, SRlan, IAS, and the composite metric AccSR.
    • Results: 7 open-source VLMs complete all settings; InternVL3.5-8B has the highest AccSR (0.426, Table 5); Drone SRact is highest at 0.642 (InternVL3.5-8B), Quadruped highest at 0.474 (Qwen3-VL-8B), Humanoid SRlan highest at 0.558 (Llama-3.2-11B) (Tables 3, 4).
    • Results: Qwen2.5-VL-3B has the highest collision-free rate (0.947/0.983) but the lowest success rate, showing a "safe but inactive" pattern; on the 200-item subset, Gemini 3.6 reaches 0.900 on Drone but only 0.205 on Wheelchair, and the authors state that closed-source capability is not consistently stronger across users.
    • Results: on 30 real trajectories the model ranking is largely preserved (InternVL3.5-8B AccSR 0.302), which the authors call preliminary evidence.
    • Authors' conclusion: current VLMs struggle to consistently adjust obstacle relevance, path selection, and safety criteria across user profiles; accessibility-aware navigation requires dedicated datasets, tasks, and metrics; the authors plan to open-source all data and code.
Read originalarxiv.org