InteractiveWorld: a verifiable human-centered environment for building better interactive MLLM assistants

Core question: how to faithfully and verifiably evaluate, without human participation, the ability of MLLMs as first-person (egocentric) AI assistants to help people in the physical world.

• Scenario and motivation: egocentric assistants on wearable devices share the user's first-person view and use natural language to guide the user through daily tasks; the authors state that real interaction experiments are expensive and hard to scale, and that real interaction data is scarce.
• Shortcomings of existing benchmarks: the authors state that existing benchmarks are almost all static—the model watches pre-recorded video to answer questions or predict when to intervene, and no matter what the model says the video does not change, so closed-loop adaptation and composition of multiple skills cannot be measured.
• Core observation of the paper: real helping is dynamic; the assistant's words change the person's actions and the frames it subsequently sees, requiring closed-loop adaptation and a combination of perception, memory, reasoning and communication capabilities.
• What the paper proposes: the authors state they propose InteractiveWorld—the first verifiable environment supporting closed-loop interaction between a human and an assistant (based on Isaac Sim, 8000+ objects, 50+ scenes); and InteractiveBench—120 human-reviewed multi-turn tasks in six categories, with success automatically determined by the simulator's final object state.
• Evaluation setup: the model under test plays the Assistant agent, and a model plays the Human agent; the Human is instructed to follow the Assistant's guidance rather than act on its own; task success is verified by whether all predefined core states are satisfied.

Egocentric AI assistant benchmarks (static, pre-recorded video)

• EgoSchema, HourVideo, EgoThink, EgoPlan-Bench, EgoLife, TeleEgo, EgoBlind, etc.—based on datasets such as Ego4D and EPIC-KITCHENS, they test long-video understanding, first-person reasoning, planning, long-term memory or blind assistance. The paper notes that they are open-loop: outputs are scored against fixed recordings and do not affect what follows.
• HoloAssist, ProAssist, ESTP—record real guidance or evaluate when a proactive assistant should say what; EgoBench introduces an LLM to simulate the user, but the visual context is still pre-recorded.
• User-in-the-loop studies (Verghese et al., 2025; PuzzleMate)—use real humans in the loop, and Verghese et al. also show that offline metrics do not reflect online performance; the paper notes that such studies have few participants, are costly and hard to reproduce.

Egocentric assistant models and systems

• MM-Ego, AlanaVLM, EgoGPT, etc. are fine-tuned on egocentric instruction data; VideoLLM-online and Dispider learn when to respond in streaming video; LifelongMemory and Ego-R1 handle long-range memory; VLaMP, Vinci and Egocentric Co-Pilot do system-level integration. The paper notes that these methods are trained and evaluated on static data or small-scale user studies, and whether they are effective in closed-loop assistance is unknown.

Robot-centric embodied simulators

• AI2-THOR, Habitat, iGibson, BEHAVIOR-1K, ManiSkill3, Isaac Lab, RoboCasa, GRUtopia, etc. provide interactive scenes; ALFRED and EmbodiedBench have models directly command robots. The paper notes that in these settings the model under test is the actor, and the human is absent or merely part of the scene; although HumanCLAW uses a simulated human body, the VLM still decides every step of the body's actions.

Interactive simulators

• In CVDN, TEACh and DialFRED, an instructor uses language to guide a follower, but the paper notes that the model is usually evaluated as the follower, and the guidance comes from replay or a rule-based oracle; in VirtualHome, Watch-And-Help, Habitat 3.0, PARTNR, CoELA, Virtual Community, HRIBench and FreeAskWorld, the AI is still the physical actor, and the human is a scripted partner or information source.

Positioning: the authors state that InteractiveWorld reverses the above roles—the AI is a bodiless assistant, and an autonomous MLLM-driven human decides how to act; in Table 4 the authors state that it is the only environment that simultaneously satisfies six properties (first-person view, human actor, guided human, natural-language dialogue, closed loop, verifiable success).

Overall approach: build the three-layer InteractiveWorld environment on Isaac Sim, letting an MLLM-driven Human agent ground the Assistant's natural-language guidance into actions; build InteractiveBench on top of it, with automatic scoring via state-verifiable rewards.

InteractiveWorld three-layer architecture (Figure 2)

• Physics layer: SMPL+H models the full body and hands (52 joints, 156-dimensional pose vector), with a camera at the head producing a first-person view; the body is kinematically driven to avoid physical failures such as falling; navigation paths and arm reachability are pre-checked before execution, and unsafe actions are rejected with a failure reason returned; 8000+ objects and 50+ scenes are imported from BEHAVIOR-1K.
• Human action layer: high-level actions are executed in two stages—automatic obstacle-avoiding navigation based on A* search (0.10 m occupancy grid), and, after arrival, manipulation with adaptive posture (standing, bending, crouching) and collision-free arm trajectory planning.
• Semantic layer: defines 36 executable action primitives ([Walk], [Grab], [PutInside], [Cook], etc.; full list in Table 5); commands specify only the action and the target; an "observe—guide—execute" loop is formed, and the system tracks object positions and states to automatically verify task success.

InteractiveBench task design

• Scale and categories: 120 manually reviewed tasks, 20 in each of six categories: Guidance, Planning, Accessibility, Decision, Prevention, Finding (Table 1); in Accessibility tasks the Human agent is deliberately not given visual input, simulating a blind or low-vision user.
• Inclusion criteria: each task must satisfy a real need for help, require the Assistant to use first-person information to help over multiple turns, and have an outcome verifiable by explicit environment states.
• Construction process (Appendix C.1, Figure 7): first generate a scene proposal, reviewed independently in two rounds; define the evaluation contract; bind scenes and objects; write the initial state, oracle action sequence, no-help sequence, and state checks; replay and verify in Isaac Sim; freeze the TaskPack. Generation used gemini3.1pro as author and gpt5.5 as reviewer, with random seed 11, temperature 0, and at most 8 revisions.

Evaluation protocol (Appendix C.2)

• Closed-loop flow: initial scene → pre-interaction warmup (the human performs predefined actions to provide context) → closed-loop interaction (the Assistant chooses silent, interrupt, or done each turn; the Human outputs action commands and replies) → terminates when the interaction limit is reached or the Assistant outputs done.
• Information boundary: the Assistant sees only the first-person view, the dialogue history, and the public task request, and cannot see simulator-internal identifiers, answers, or scores; the Human is forbidden to infer answers on its own or repair the Assistant's plan, and each of the six task categories has specific restrictions (e.g., in FIND the Human cannot itself infer the remembered location).
• Success determination: each task predefines a set of core states (e.g., "the pizza is in the oven"); at task end the actual environment state is checked, and success requires all to be satisfied; satisfying the goal does not automatically terminate the task, preventing models from succeeding by enumeration and luck.

Metrics

• Pass and MCR: Pass is the percentage of fully completed instances; MCR (Mean Completion Rate) is the average of the per-instance proportion of goals completed, averaged over the 20 task families (Appendix C.3, Equation 6).
• Capability score: nine capabilities (C1–C9, Table 6) are aggregated with equal weight by capability evaluation points (capability atoms); only clean_pass counts as a pass.
• Efficiency: computed only on strictly successful episodes, as the ratio of the model's action steps to the oracle's shortest legal step count for the same task.

Experimental setup

• Models under test (Assistant): 12—Claude Fable 5.1, Claude Opus 5, Claude Opus 4.8, GPT-5.6 Sol, GPT-5.6 Luna, GPT-5.5, Muse Spark 1.1, GLM 5.3 Flash, Kimi K3, Qwen 3.8 Max, Gemini 3.1 Pro, DeepSeek V4.1 Flash.
• Human agent: fixed to Gemini 3.1 Pro in the main experiments; the Human agent is changed in the Section 4.2 ablation.
• Benchmark: InteractiveBench, 120 tasks, 20 in each of six categories.
• Input settings: visual input capped at 32 frames per turn, with 4 frames sampled per action; when over the limit, the most recent 8 frames are kept and 24 frames are sampled uniformly from older history.
• Metrics: Pass (full completion rate), MCR, capability score (C1–C9), action efficiency (model steps/oracle steps, computed only on passed tasks).
• Validity check: the authors verified that the same Human agent with an Oracle Assistant reaches 100% Pass, and 0% with no help.

Main results

According to Table 2 (Pass %, Human agent = Gemini 3.1 Pro):

Swipe horizontally to read the table

Model	AVG.	Guidance	Planning	Access.	Decision	Prevention	Finding
Claude Fable 5.1	41.7	20.0	25.0	30.0	85.0	50.0	40.0
Claude Opus 5	33.3	15.0	25.0	25.0	70.0	30.0	35.0
Muse Spark 1.1	31.7	20.0	25.0	35.0	55.0	30.0	25.0
GPT-5.6 Sol	30.8	20.0	25.0	30.0	70.0	15.0	25.0
GPT-5.5	30.0	10.0	15.0	45.0	65.0	25.0	20.0
GLM 5.3 Flash	30.0	15.0	20.0	50.0	50.0	20.0	25.0
Claude Opus 4.8	28.3	0.0	20.0	30.0	50.0	35.0	35.0
DeepSeek V4.1 Flash	12.5	0.0	5.0	25.0	35.0	0.0	10.0

(Kimi K3 24.2, Qwen 3.8 Max 22.5, Gemini 3.1 Pro 20.8, and GPT-5.6 Luna 20.0 are omitted; all are listed in Table 2.)

• The authors state that the strongest model, Claude Fable 5.1, averages only 41.7%, and all other models are no higher than 33.3%, so closed-loop assistance is far from solved (Finding 1).
• The authors state that models reach up to 85.0% on Decision, which requires a single judgment, but perform poorly on Guidance (20.0%) and Planning (25.0%), which require continuous tracking and replanning; the difficulty lies in maintaining the closed loop (Finding 2).
• The authors report that GLM 5.3 Flash and Claude Fable 5.1 have close average capability scores (58.2% vs 59.2%), but communication scores of 69.8% vs 79.6%, and 11.7 percentage points fewer task passes; the authors argue that closed-loop success is limited by the weakest capability in the chain (Finding 4, Table 3).

Error type analysis (Figure 4, Appendix D.1)

• Method: for each valid failure episode, the earliest critical error is manually traced back and classified into six types (R1 visual misreading, R2 loss of state tracking, R3 rule/causal reasoning error, R4 planning error, R5 unexecutable instruction, R6 wrong intervention timing), with only one primary label counted per failure.
• Results: across models, perception (R1) and memory (R2) together account for 70.4%–85.7% of first errors; for Gemini 3.1 Pro, R1 accounts for 69.8%, and for Kimi K3 memory errors exceed perception errors (39.2% vs 33.8%).
• Authors' interpretation: the authors believe current models lack the basic visual and memory capabilities required for interaction, and a single misreading cascades into all subsequent instructions (Finding 3).

Efficiency, memory augmentation, and Human agent ablation

• Action efficiency (Figure 3): the authors state that almost all models show action redundancy relative to the Oracle, reaching 2.35×–3.66× on Accessibility and Guidance; Claude Fable 5.1 is 6.75× on Accessibility and 1.20× on Prevention, and Kimi K3 is 1.00× on Finding and 3.66× on Guidance; the authors argue that efficiency is a task-dependent rather than a fixed model property (Finding 5).
• Memory augmentation (Figure 5 right, Appendix D.2): after adding a hidden Memory field to the Assistant, three models improve Pass by 5–15% on Prevention and Finding, but two models drop 5% on Planning, with mixed effects in the remaining categories; the authors argue that simple memory augmentation is insufficient to support reliable long-horizon assistance.
• Human agent effect (Figure 5 left): four models are crossed pairwise as Assistant/Human, comparing MCR on Guidance, Planning, and Accessibility; the authors state that Assistant rankings are relatively stable (Fable 5.1 always first, GPT-5.6 last), and all four Assistants achieve their highest MCR when Gemini 3.1 is the Human.

Other ablations and analyses

• Static comparison (Appendix D.3, Table 13): the same 120 tasks are converted to static question answering and tested on 8 models; static accuracy is generally higher than dynamic (e.g., Claude Fable 5.1 static Planning 65% vs dynamic 25.0%), and the authors state that static evaluation overestimates model capability.
• Task duration effect (Appendix D.4, Figure 9): family-adjusted slope −0.431 (95% CI −58.1 to −28.9, one-sided permutation p < 0.0001); the authors note this is correlational rather than causal, and that some labels were reconstructed from saved traces and the results are regarded as preliminary.
• Human verification (Appendix D.5, Table 14): 3 real humans as executors, 36 tasks; changing the executor changes each Assistant's number of passes by at most 3/36 (8.3 percentage points); the average task-level agreement between the Gemini 3.1 executor and real humans is 84.3%, and between real humans 85.2%; the authors state this provides preliminary support for using a model Human agent as a proxy for real humans, and note that the ranking of Muse 1.1 and GPT-5.6 Sol changes with the executor.
• Proactive behavior: the authors observed models proactively asking the Human to look around, turn over packaging to read labels, and answer questions about spatial relations, among other behaviors not prompted by the instructions.

Limitations and future directions noted by the authors

• RL training platform: the authors plan to build a reinforcement learning platform based on InteractiveWorld's verifiable state rewards to train interaction capabilities (Appendix E).
• More realistic Human agent: the authors plan to make the Human agent more human-like through training and other means, and to simulate users with different personalities and skill levels (Appendix E); Section 4.2 also mentions that future work will explore behavior calibration to align the simulated Human with real users.
• More real scenarios: the authors plan to leverage InteractiveWorld's capability to support scanned real scenes to introduce more real environments and situations requiring an assistant (Appendix E).
• Memory mechanisms: in Section 4.2 the authors state that simple memory augmentation is insufficient and that future work needs to train native dynamic interaction memory mechanisms.
• Caution about the duration analysis: in Appendix D.4 the authors note that the relationship between task duration and success rate is correlational rather than causal, and that some labels were reconstructed from saved traces, so the results are regarded as provisional.
• Ranking invariance: in Appendix D.5 the authors note that the ordering of Muse 1.1 and GPT-5.6 Sol changes with the executor, so they do not claim the full ranking is invariant.

Scope of experiments

• Models under test: the main evaluation uses 12 frontier MLLMs as Assistant; the static comparison and duration analysis each test 8 models (Appendix D.3, D.4).
• Human agent: fixed to Gemini 3.1 Pro in the main experiments; the ablation covers 4 models paired with each other (Figure 5); human validation has only 3 participants and 36 tasks (6 per category), and the authors note that these comparisons reuse the same tasks and are not independent samples or statistical equivalence tests (Appendix D.5).
• Task scale: 120 tasks, 20 each in six categories; scene assets come from BEHAVIOR-1K (44 layouts passed task admission, Appendix B.1.3).
• Memory augmentation ablation: only 3 models (Fable 5.1, GPT-5.6-sol, Gemini 3.1 Pro, Figure 5 right).
• Not reported in the paper: the number of repetitions in the main evaluation (how many runs per model per task); the number of human annotators and the agreement for the error-type analysis.

An evaluation/benchmark paper: the authors propose the InteractiveWorld environment and the InteractiveBench benchmark, answering "can today's MLLMs truly help people in the physical world".

• Problem: evaluating egocentric AI assistants requires real human interaction and is costly; existing benchmarks are static video question answering, where model outputs do not affect what follows, so closed-loop collaboration cannot be measured.
• Method: InteractiveWorld is built on Isaac Sim in three layers (physics, human actions, semantics); an MLLM-driven Human agent grounds the Assistant's natural-language guidance into 36 action primitives; InteractiveBench contains 120 human-reviewed tasks (20 each in six categories), with success automatically verified by the simulator's final object state; the authors verified that an Oracle Assistant reaches 100% and no help gives 0%.
• Main results: among 12 frontier MLLMs, Claude Fable 5.1 has the highest average Pass at only 41.7%, and all others are no higher than 33.3%; the Guidance and Planning categories reach at most only 20.0% and 25.0%, and the Decision category reaches at most 85.0% (Table 2).
• Errors and efficiency: among each model's first errors, perception and memory errors account for 70.4%–85.7% (Figure 4); action redundancy relative to the Oracle reaches 2.35×–3.66× on Accessibility and Guidance (Figure 3).
• Other findings: the static question-answering version of the same tasks has noticeably higher accuracy, and the authors state that static evaluation overestimates capability; in validation with 3 human participants, the task-level agreement rate between the Gemini 3.1 executor and humans is 84.3%, close to the 85.2% between humans.
• Authors' conclusion: the authors believe current MLLMs are still far from reliable interactive assistants, and that perception, memory and communication are key bottlenecks; future plans include building an RL training platform, making the Human agent more realistic, and introducing more real scenarios.