Skip to content

Prospects for Agentic Robotics: A New Era of Robotic Systems

Envisioning Agentic Robotics: A New Era of Robotic Systems

Hugging Face Daily Papers· arXiv:2611.18359· Grid Robotics, :, Qingfeng Lan, Duo Li, Siyao Li, Yiheng Lyu, Shijian Lu, Zhenghao Qi, Shizun Wang, Xinchao Wang, Ming Yang, Zhiqin Yang, Zuhao Yang, Tao Yu·Indexed by this site · 4 days ago· Originally publishedOpinion· Its place in the scan record →

Quick read

Other

Based on the paper PDF (AI-generated); refer to the original

Grid Robotics proposes an agentic robotics framework: a foundation model plus an embodied harness forms the Embodied System 2 decision core, along with a nine-dimensional evaluation framework and an L1–L4 responsibility roadmap.

Questions
After the release of GPT-6 Astra, frontier models with a harness match task-specific policies on multiple embodied benchmarks, but the community lacks a unified definition of agentic robotics, a reliable evaluation framework and a research roadmap, and results reported by different studies are difficult to compare.
Methods
Proposes a three-layer functional architecture: Embodied System 2 (foundation model plus embodied harness) is responsible for intent understanding, decision-making, orchestration and verification, S1 provides skills and tools, and S0 is responsible for real-time control; it also gives a nine-dimension, 53-subdimension evaluation framework, nine evaluation principles and an L1–L4 delegable-work ladder.
Experiments and results
The paper ran no new experiments but instead aggregates published results: Astra reaches 92.1% on RoboMME, but only 19.0% on RobotWorld; RT-Safe real-time execution lowers the safety success rate from 19.8% to 0.7%; on RoboHarm, the text refusal rate of 72% drops to 18% on real machines.
Limitations and future work
The authors point out that current evaluations have insufficient coverage in responsibility, space and time, with no benchmark workflows or roles lasting several days; the validity of simulation results for real robots is unverified; comparability under continuous time, trustworthy acceptance of complex tasks, and scaling of real-robot evaluation remain open problems.
Experimental setupGPT-6 Astra、GPT-5.6-Sol · RoboDojo、RoboLab-120 et al. · success rate、safe success et al.
Evaluated models
GPT-6 AstraGPT-5.6-Sol
Benchmarks
RoboDojoRoboLab-120LIBERO-PRORoboMMELADDERRobotWorldRoboQuestHumanCLAWEmbodiedSkillBenchRT-SafeRoboHarmEmbodied First Exam
Metrics
success ratesafe successrefusal rateobservation agelatencycost
AI summaryThe paper proposes an Embodied System 2 composed of foundation models and an embodied harness as the decision-making core of agentic robots. The authors state that the release of GPT-6 Astra marks the rise of this paradigm, and that within weeks frontier models with agentic harnesses matched or exceeded task-specific policies on multiple embodied benchmarks. Its evaluation framework measures both model capability and closed-loop system performance, and the roadmap points toward continuous autonomy from executing individual action primitives to taking on standing roles.

The paper proposes an Embodied System 2 composed of foundation models and an embodied harness as the decision-making core of agentic robots. The authors state that the release of GPT-6 Astra marks the rise of this paradigm, and that within weeks frontier models with agentic harnesses matched or exceeded task-specific policies on multiple embodied benchmarks. Its evaluation framework measures both model capability and closed-loop system performance, and the roadmap points toward continuous autonomy from executing individual action primitives to taking on standing roles.

Paper analysis

6 questions; open one to read the full answer

  1. What problem does this paper attempt to solve?

    Establish a unified definition, evaluation framework and research roadmap for agentic robotics, with the core being a decision layer composed of a foundation model plus an embodied harness.

    Core problem: agentic robotics lacks a shared definition, a reliable evaluation framework and a research roadmap, and results reported by different studies are difficult to compare.

    • Problem setting: After the release of GPT-6 Astra, frontier models with an agentic harness match or exceed task-specific policies on many embodied benchmarks, which the authors regard as marking a paradigm shift from policy-centric robotics to agentic robotics.
    • Existing shortcomings: The authors point out that evaluations often do not state whether they test the base model or the complete system, rarely disclose the full protocol (observations, budget, demonstrations), and many evaluations pause the world while the model thinks; for example, on RoboDojo Astra's reported success rate ranges from 22% to 85% depending on the harness and protocol.
    • Core claim: The authors argue that Embodied System 2—realized jointly by a foundation model and an embodied harness—is the decision-making core of agentic robotics; its core challenge is to turn general reasoning into effective and verifiable actions under physical constraints such as limited reaction time, mechanical limits, environmental change and irreversible consequences.
    • Paper outputs: Proposes an S2/S1/S0 reference architecture, a nine-dimension evaluation framework (distinguishing model capability from closed-loop system performance), a transfer analysis between digital agents and physical agents, and a research roadmap from execution primitives (L1) to taking on sustained roles (L4).
    • Nature of the paper: This is a position and framework paper that collects and analyzes published results and reports no new experiments run by the authors themselves.
  2. What related research exists?

    Related work falls into four groups: surveys and definitions, language-model robot agents, hierarchical/dual-system models and embodied reasoning models, and harnesses and benchmarks.

    Surveys and position papers

    • Bommasani et al. (2021) proposed the foundation model concept; subsequent surveys organize robotics and VLA research by component (perception, decision-making, control) or by model role.
    • Wang et al. (2026b) code 2,343 systems by where foundation-model decision-making occurs; Qin et al. (2026a) organize the survey around models, data, environments and harnesses; Chen et al. (2026d) find that of 64 benchmarks, only 5 evaluate expectation and 3 evaluate verification.
    • Authors' positioning: Like these works, they define S2 functionally, but add testable judgment criteria and count the harness as part of Embodied S2 (Table 12).

    Language-model robot agents

    • Early systems (2022–2025): use language models to decompose tasks, combine affordances, compose perception and control in code (Code as Policies, Liang et al., 2023), and close the loop with feedback (Inner Monologue, Huang et al., 2023b).
    • 2026 systems: HexaShell uses an optimizer agent to write tools (Gao et al., 2026b); uni-robot-agent and others combine primitives with VLA policies (Si et al., 2026); general agents directly drive physical robots (Jia et al., 2026).

    Hierarchical/dual-system models and embodied reasoning models

    • The upper modules of Helix (Figure AI, 2025b) and GR00T N1 (NVIDIA et al., 2025c) output only latent representations, so the authors do not classify them as S2; π0.5 (Black et al., 2025a) outputs subtasks first and then actions, and the authors consider its subtask stage to implement part of S2.
    • Embodied reasoning models include Gemini Robotics-ER, Cosmos-Reason1, Embodied-R1/R1.5, RynnBrain, MiMo-Embodied, HY-Embodied, Vesta, ACE-Brain, PhysBrain, etc.; the authors consider that these reports mostly evaluate offline question answering and spatial skills, and their value to the S2 loop needs to be verified at the system level.

    Harnesses, benchmarks and digital-agent precedents

    • Digital precedents: ReAct, Toolformer, Reflexion, SWE-agent (Yang et al., 2024) show that the agent–computer interface affects performance; physical harnesses include Thea, DynaHarness, RegenHarness, SCOPER, etc.
    • Comparison benchmarks cited in the paper include RoboDojo, RobotWorld, RoboMME, LIBERO-PRO, LADDER, RoboQuest, HumanCLAW, EmbodiedSkillBench, RT-Safe, RoboHarm, etc.; compared policies include trained policies such as π0.5 (Table 8).
    • Authors' positioning: This paper treats the harness as a component of Embodied S2 rather than an add-on layer outside the model, and requires each result to be accompanied by a harness card for attribution.
  3. How does the paper address this problem?

    Defines Embodied S2 and gives judgment criteria, designs a five-component harness, and proposes a nine-dimension evaluation framework, nine evaluation principles and an L1–L4 roadmap.

    Overall approach: decompose agentic robotics into three functional roles, with Embodied System 2 (foundation model + embodied harness) as the decision-making core, S1 providing skills and callable modules, and S0 providing the body interface and real-time control; on this basis, build the evaluation framework and roadmap.

    Reference architecture and S2 judgment criteria

    • Three-layer functional roles: S2 interprets intent, reasons about goals and constraints, orchestrates skills and tools, and verifies results; S1 contains VLA/WAM policies, perception and motion planning modules, and programs generated by S2; S0 performs real-time control (typically 50 Hz–1 kHz). The decomposition is functional: one model can serve multiple roles.
    • Judgment criteria: A component acts as S2 if and only if it makes two kinds of decisions—how to proceed (choosing subgoals, skills or replanning, etc.) and judging the result (determining from the previous step's evidence whether the goal was achieved)—and both kinds of decisions can be identified externally (interface exposure or controlled substitution experiments).
    • Three execution paths (Figure 4): hierarchical orchestration (S2→S1→S0), direct control (S2 via the action interface to S0), and code-as-policy (S2 writes code that acts as S1); a system can mix them within a single task.
    • Real-time event monitor: Independent of the three-layer system, it monitors the user, body and scene at high speed, can interrupt S2, and directly triggers S0's protective responses according to predefined safety rules.

    Composition of the embodied harness

    • Five core components (Table 1): context manager (assembles world, user and body state), tool interface, planner, memory, verifier; three enhancements: safety gate, budget control, self-improvement.
    • Execution contract: Four kinds of contracts (observation snapshot, capability contract, call record, evidence bundle) define the lifecycle of each physical call: propose → admit → dispatch → execute → confirm the terminal state → verify the effect → commit progress, including branches for rejection, failure, cancellation and unknown results.
    • Three feedback loops: sensorimotor loop (S1/S0), planning loop (S2), evolution loop (decides which capabilities to use on the next run); the authors consider that conflating the three would obscure the source of reactivity or improvement.

    Physical constraints and capped scaling

    • Capped scaling: The authors state that frontier models follow monotonic scaling (more reasoning tokens for better answers, longer latency), whereas Embodied S2 needs to improve capability within a fixed decision deadline (Figure 7; the authors note the curves are illustrative).
    • Latency data (Figure 8): S0 controls in 1–20 ms per step; S1 controls skills in 5–200 ms; Astra directly issuing actions takes 18.56 s per inference and 39.86 s on average to generate motion references; the authors state that when Astra is used directly as S2, each decision is about 40 to 8,000 times a single update of S1 skill control.
    • Observation age: Δ tobs = teffect − tcapture, i.e., the interval from the moment the observation on which a decision is based was captured to the moment the action takes effect; the authors consider it closer to physical response than model inference latency.

    Evaluation framework

    • Two levels: Model level (the model answers alone, usually single-turn without tools) tests the model-side parts of knowledge, understanding and reasoning, and safety; system level (the model acts in a closed loop through the harness) tests closed-loop reasoning, tool use, memory, cross-embodiment, safe execution, efficiency and cost.
    • Nine dimensions and 53 subdimensions (Figure 9): knowledge, understanding, reasoning, tool use, memory, cross-embodiment, safety, efficiency, cost; of these, 29 subdimensions are embodiment-related; the authors require reporting a profile by dimension rather than aggregating into a single score.
    • Nine evaluation principles: separate model and system, treat the action interface as a controlled variable, run on the world clock (report paused and real-time separately), disclose the observation contract, verify completion independently, quantify uncertainty, control and sweep the budget, validate on real robots, and fix the level claim conditions in advance; and require each result to be accompanied by a harness card.

    Roadmap

    • L1–L4 ladder (Figure 12): L1 primitives (seconds), L2 tasks (minutes), L3 workflows (hours), L4 roles (days and above); each level builds on the one below, and progress is measured by the scope of responsibility the robot can independently take on.
    • Four research directions: embodied reasoning under physical constraints, trustworthy persistent control across tasks and spaces, efficient adaptation across environments and embodiments, and continuous improvement from physical experience (including distinguishing adaptation, continual improvement and recursive self-improvement).
  4. What experiments did the paper conduct?

    The paper conducts no new experiments, but instead compiles published results (Table 8) and analyzes them under seven conclusions, with varying strength of evidence.

    Experimental setup

    • Nature of the paper: The authors ran no new experiments; Table 8 collects results published as of October 2026, and the authors state that they are reported as-is, not reproduced, and that the settings of different rows differ and are not mutually comparable.
    • Models tested: General-purpose models serving as the S2 foundation model, mainly GPT-6 Astra; also GPT-5.6-Sol (EmbodiedSkillBench), nine VLMs from HumanCLAW, eight VLMs from RT-Safe, and six frontier models from RoboHarm (the paper does not list their names individually).
    • Comparison targets: Trained policies on the leaderboards of the respective benchmarks, such as the 40 public policies on RoboDojo and π0.5 on LIBERO-PRO (53.0%, with the authors noting that the evaluation setting differs).
    • Metrics: Mainly success rate, plus safety success rate, refusal rate, collision rate, tokens and cost, and latency.
    • Evaluation method: The paper does not unify the evaluation method; its principles require success to be judged by a checker that can access the system's unobservable ground truth, and it cites research finding that agents claim success early and that VLM judges are biased toward reporting success.
    • Sample size and repetitions: These vary by source, e.g., RoboMME is 800 test episodes in a single run; SenseNova reports 5 episodes per task; the authors criticize that many headline results come from a single run or a small number of seeds.

    Main results

    According to Table 8 (settings differ across rows and are not mutually comparable; several simulations pause while the model thinks):

    Swipe horizontally to read the table

    BenchmarkS2 modelS1 componentResult
    RoboDojo-SimGPT-6 Astranone22.48%, higher than the 40 public policies at the time
    RoboDojo-Sim official evaluationGPT-6 Astraagent-written tools46.9%, top of the leaderboard at release
    RoboLab-120GPT-6 Astra direct/agent sessionnone66.3% / 74.8%; best leaderboard policy 51.0%
    LIBERO-PROGPT-6 Astramotion primitives, segmentation, π0.594.0%; π0.5 is 53.0% (different setting)
    RoboMMEGPT-6 Astranone92.1%; best leaderboard entry 60.8%
    LADDERGPT-6 Astranone62.0%; highest learned-policy system 29.6%
    RobotWorldGPT-6 Astranone19.0%; 63 tasks unsolved by all five models
    RoboQuestGPT-6 Astranone23.2%; fine-tuned π0.5 almost never succeeds
    • The authors state that general models with a harness are competitive on narrower simulation suites, but far from solved on broad benchmarks that are cross-domain or require active exploration (HumanCLAW best is 16.8%).
    • The authors state that high scores come from narrower suites, and that in some comparisons the model received information the policies did not have (e.g., written task recipes); under the same interface, the previous-generation model scores only 0.88% on RoboDojo, indicating that the results are model-specific.

    Sensitivity to interface, real-time execution and physical actions

    • Interface effects (EmbodiedSkillBench): Astra native control 92%, skills-only 61%, decide-on-demand whether to use skills 85%; for GPT-5.6-Sol it rises from 11% to 63%; the authors state that skills help only when they exceed the model's existing capabilities, and skills-only execution also reduces token usage for weaker agents by 58%–85%.
    • Real-time execution (the hardest RT-Safe route): real-time execution raises collisions to 12.3 times, lowers safety success rate from 19.8% to 0.7%, and leaves task completion essentially unchanged; the authors state that pausing the world masks latency, cost and risk.
    • Text-to-action refusal gap (RoboHarm real robots): the refusal rate of six models is 72% for text requests and 18% for real-robot end-effector control trials; the authors state that refusal learned from text transfers only partially to actions, and safety must be measured in the action loop.

    Effects of harness and protocol

    • Harness changes: With the model fixed, changes to the verifier, memory, interface and post-call decision-making move success rates by about 9–35 percentage points across four studies; in a fifth, restricting to skills-only moves it by up to 52 percentage points (Han et al., 2026).
    • Protocol differences: On RoboDojo, Astra's reports range from 22.48% (one seed, wiki recipe, direct end-effector control) to 85.0% (about one page of guidance per task); after one study re-scored under RoboDojo's native step limit, one system's success rate dropped from 47.1% to 1.0% (97 of 99 successes exceeded that limit).
    • Hidden verifier: After tool returns no longer reveal success information, the average scores of two systems on 10 RoboDojo tasks and 50 episodes dropped from 83.2 to 63.5 and from 42.0 to 26.1 (OpenRUA Team, 2026).

    Other ablations and analyses

    • Reasoning budget: In Astranomy, raising effort from medium to extra-high raises success rate only from 72.7% to 74.5%, with 2.6 times the latency and 2.2 times the cost (Yang et al., 2026d).
    • Cost: In Astranomy, the agent Astra uses 30.9M tokens per successful episode ($36.40), versus 160k tokens ($1.96) for direct prompting.
    • Memory: Frozen Astra places the entire episode in context and succeeds on 92.1% of the 800 RoboMME test episodes in a single run, averaging 18.3 s per call (simulator paused).
    • Spatial understanding: Astra at the low reasoning setting scores 59.8 on VSI-Bench, versus 45.4 for the best model in the original evaluation and 79.2 for humans (400-question subset).
    • Abstention ability: On AbstainEQA, the best frontier model identifies 42.79% of unanswerable questions, versus 91.17% for humans.
    • Safety judge: In one study, an LLM judge misses nearly 83% of unsafe trajectories (Zhan et al., 2025).
  5. What are some points that can be further explored?

    The authors list four categories of limitations, including evaluation coverage and simulation validity, and propose four research directions; the evidence mostly comes from single runs.

    Limitations noted by the authors and future directions

    • Insufficient evaluation coverage (Section 4.4): the benchmarks reviewed mostly test single-episode primitives and tasks within a work unit or room; the authors state that no benchmark tests workflows that share resources and deadlines, or roles lasting several days.
    • Simulation validity unverified (Section 4.4): the authors state that they do not know whether the rankings of the S2 foundation model and harness in simulation hold on real machines, nor which properties of the simulator determine this.
    • Comparability under continuous time (Section 4.4): when the world is not paused, runs are also not repeated in timing; how to control timing differences at a cost tolerable for real machines is an open problem.
    • Trustworthy acceptance of complex tasks (Section 4.4): VLM judges are biased toward reporting success when evidence is ambiguous; how to obtain acceptance judgments with known error rates for workflows and roles is an open problem.
    • Scaling real-machine evaluation (Section 4.4): real-machine trials are slow and expensive, and many reports have only a small number of trials per task; how to obtain statistically reliable evidence over larger spaces, longer times, and broader responsibilities is an open problem.
    • Four research directions (Section 6.2): embodied reasoning under physical constraints (treating capped scaling as a training and system objective), persistent verifiable control across tasks and spaces, efficient adaptation across environments and embodiments, and continuous improvement from physical experience; the authors state that the gains from continuous improvement cannot yet reliably accumulate, and that physical recursive self-improvement (Physical RSI) has not been demonstrated according to the cited review.

    Scope of experimental coverage

    • The paper itself ran no experiments; the published results relayed in Table 8 cover 12 benchmarks/settings, and the authors state that they did not reproduce them and that the settings differ across rows and are not mutually comparable.
    • The models tested are mainly GPT-6 Astra; there are also GPT-5.6-Sol, nine VLMs of HumanCLAW, eight VLMs of RT-Safe, and six frontier models of RoboHarm; the paper does not list the names of the latter three one by one.
    • Real-machine evidence includes the refusal-rate comparison of RoboHarm and one real-machine evaluation (task duration about 100 times that of a human); most results are simulation, and several simulations pause while the model is thinking.
    • The controlled studies across model × harness conditions are small in scale: at most four models and a small number of conditions on the same task (Ge et al., 2026; Wei et al., 2026a; Han et al., 2026).
    • The paper reports no statistical tests on the relayed results; the Robot Intelligence Index it cites reports, for each model, a ranking interval under 90% resampled reruns.
  6. Summarize the main content of the paper

    A position paper: defines Embodied S2 as the decision core of agentic robotics, with a nine-dimensional evaluation framework and an L1–L4 responsibility roadmap.

    A Grid Robotics position paper addressing the lack of a unified definition, incomparable evaluations, and the absence of a roadmap for agentic robotics after the release of GPT-6 Astra.

    • Problem: different reports differ greatly in harness, protocol, and temporal semantics; for example, reported success rates for Astra on RoboDojo range from 22% to 85%, making cross-study comparisons meaningless.
    • Method: proposes an S2/S1/S0 reference architecture; Embodied S2 is realized jointly by a foundation model and an embodied harness (five components: context manager, tool interface, planner, memory, verifier), with testable criteria for judging S2 by two types of decisions; under physical constraints it proposes capped scaling, i.e., improving capability within a fixed decision time limit.
    • Evaluation framework: two levels, model-level and system-level, with nine dimensions and 53 sub-dimensions (29 embodiment-related), and nine evaluation principles (including running on world clock, independently verifying completion, and public observation contracts), with a harness card attached to each result.
    • Main evidence (compiled from published results with differing settings): Astra reaches 92.1% on RoboMME and 94.0% on LIBERO-PRO, but only 19.0% on RobotWorld and a best of 16.8% on HumanCLAW; RT-Safe real-time execution lowers the safety success rate from 19.8% to 0.7%; on RoboHarm the refusal rate drops from 72% in text to 18% on real robots; in EmbodiedSkillBench, the skill interface alone cuts Astra from 92% to 61%.
    • Roadmap: progress is measured by delegable responsibility—L1 primitives, L2 tasks, L3 workflows, L4 roles; the authors believe current systems can execute primitives and tasks, are just beginning to touch workflows, and have not yet reached the role level.
    • Authors' conclusion: most principles of digital agents transfer, but the assumptions their reliability depends on do not hold in the physical world; the authors call on the community to prioritize building reliable evaluation, improving time-limited reasoning efficiency, and standardizing reporting of model and harness configurations.
Read originalhuggingface.co