CarryOn: A dual-loop framework for personalized and reliable robot assistance

Everyday assistance must decide what to do when preferences and scenes change, and protect or repair execution after dangerous contact or physical failure.

• Motivating scenario: The authors believe general VLAs can already execute manipulation instructions across tasks, but everyday assistance also needs to know what to do for the current user and environment, and how to complete tasks when physical conditions change. The example in Figure 1 is a user saying only "Serve me food"; the system must carry forward an earlier vegetarian preference and continue delivering carrots to the bowl after obstacles or grasp failures.
• Existing gaps: The authors state that preference learning or personal item recognition can provide context, but real context changes, and stored memory needs revision. Hierarchical VLAs and concurrent embodied harnesses mainly target out-of-distribution performance and do not connect "whether it is appropriate for the current user" with dangerous-contact intervention and failure repair during execution.
• Authors' setting: The same request can correspond to different actions; plans must be rewritten when observation overrides memory; dangerous contact is handled before execution, already-occurred failures must be repaired, and failure experience is kept for future recovery.
• This paper's approach: The authors propose CarryOn, a dual-loop harness wrapped around a frozen VLA. The planning loop maintains revisable user-level and scene-level memory and dispatches subtasks; the execution loop filters actions predicted to have dangerous contact, and after failures invokes reusable control recovery skills that can be revised via offline reflection.

Persistent personalized context

• TidyBot (Wu et al., 2023) learns placement preferences that generalize to subsequent requests.
• Bring My Cup (Lee et al., 2025) uses reference image memory and visual prompting for personalization.
• Embodied personalization work studies cross-interaction memory; language agents (MemGPT; Park et al., 2023) retrieve and integrate experience for subsequent decisions. The authors claim that unlike these works, CarryOn connects persistent preferences with revisable scene knowledge: user memory constrains task selection, and scene memory guides subsequent actions.

Harness for policy execution

• Code as Policies (Liang et al., 2023) generates programs over perception and control APIs.
• Language model planners ground instructions through affordances, feedback, and multimodal observations; uncertainty-aware planners decide when to act or request information.
• Hierarchical VLAs connect instructions to visuomotor execution. RoboHarness (Huang et al., 2026) orchestrates heterogeneous policies with memory; Harness VLA (Zhang et al., 2026) combines frozen contact-rich primitives with parsing operations and memory-guided retries. The authors compare that CarryOn still has the VLA execute operations and couples persistent personalization with action filtering and physical repair, with the two loops sharing the current task.

Post-failure recovery and adaptation

• Vision-language models provide semantic failure feedback. FailSafe (Lin et al., 2025) trains failure reasoning and recovery from generated failure recovery cases; RecoveryChaining (Vats et al., 2025) learns local policies that hand control back to the nominal controller; Zetta (Ding et al., 2026) evolves runtime critics and recovery skills outside a frozen policy.
• Language agents accumulate experience through memory and reusable skills (Reflexion, ExpeL, Voyager).
• The authors claim this paper restricts revision to control-based recovery: offline reflection modifies supported parameters and policies and retains successive versions; the VLA and geometric shield remain fixed.

Geometric protection in manipulation

• Latent-space safety filters perform reachability analysis in learned world models; AnySafe adapts such filters to parameterized constraints.
• VLSA (Hu et al., 2025) adds a barrier function safety layer; Code-as-Monitor (Zhou et al., 2025) generates executable constraints to monitor failures.
• The authors claim this paper's shield predicts the spatial response of policy actions and filters by the minimum distance between the gripper or carried object and task-relevant obstacles, checks before contact, and does no online learning.

Baselines and benchmarks

• Safety comparisons are conducted on LIBERO-Safety, with baselines including original and safety-finetuned π0.5, adapted VLSA, and extensions to OpenVLA-OFT and VLAct.
• Recovery comparisons are conducted on LIBERO-Object perturbations and LIBERO-Recovery, with baselines including Qwen3-VL-OFT, CaP-X, and Harness VLA.
• Planning evaluation includes real-robot dispatch tasks and RoboLab endpoint selection and memory revision.

The authors position this paper at the connection between personalized decision-making and physical execution feedback, rather than merely improving VLA task performance in out-of-distribution scenes.

CarryOn把冻结VLA放在双环运行时里：规划环决定对当前用户和场景做什么，执行环在接触前过滤动作、失败后用控制技能修复。两环共享任务目标，视觉反馈决定推进计划还是进入恢复。经验分两种：供以后规划的上下文，以及供技能库逐版修订的恢复经验（Figure 2）。

规划环：持久且可修订的上下文

• 输入与输出：给定请求c和可选参考图像，规划器结合用户记忆Mu、场景记忆Ms和可用技能描述，产出子任务。每个子任务含指令ℓi、提供技能的端点，以及可选的前序依赖。端点包括操作工位和信息采集工具，因此计划可以先观察或查询外部信息再决定操作。
• 两类记忆：用户档案存偏好、习惯和用户物体的参考图像，每次请求注入规划上下文；规划器被指示只使用请求用户的档案。场景记忆按地点存物体清单和描述，用于选择端点和判断操作前是否需要观察。
• 修订：对话中，规划器被指示保存跨会话仍成立的偏好，一次性要求留在当前对话；计划结束后，后台summarizer按同一标准补写尚未记录的用户事实。用户修改或撤回偏好时，对应条目被替换或删除。观察揭示目标，或发现记忆位置为空、过时时，必须做场景更新；规划器在下一计划中写入update_scene。若漏掉该操作，计划校验器把计划退回修订。附录B给出存储格式和修订示例，附录F给出完整提示词。

主动几何防护

• 位置：shield位于策略和控制器之间。对每个动作块，它用当前RGB-D、目标与目的地描述构造障碍集，预测夹爪和携带物运动，再过滤动作。输出是可执行前缀；若第一步被挡，则短退后再用新观察重估。
• 障碍：抓取和放置必须接触目标与目的地，因此不把所有场景点都当障碍。VLM从指令推断目标与目的地描述，场景构建器用文本提示分割关联掩码，并把掩码深度投到三维。接触规则把允许的目标、支撑面和目的地几何排除出相应物体的障碍集；未分配几何仍是障碍。附录C给出安全评测中的接地与接触规则。
• 预测与过滤：仅有动作增量不足以描述实际位移。对接下来H个动作，shield用已执行运动历史和短时域响应模型预测末端位姿。平移用临界阻尼近似，p̈=ω_n²(p_goal−p)−2ω_nṗ，其中p为末端位置，pgoal由步初位置加上缩放后的平移动作得到，ωn设定响应速率。前缀长度为g=min({k∈{0,…,H−1}:min_a(d_a,k−δ_a)≤ 0}∪{H})，即在任一物体到其障碍的距离降到阈值及以下之前可放行的动作数。g=0时扣住候选动作并短退。只有实际执行的动作才写入历史。
• LIBERO-Safety参数：Table 7给出操作空间位姿控制，kp=150，阻尼比1，控制周期0.05 s，ωn=√150≈12.25 s−1，预测时域H=5，干预阈值δ=6.67 cm。Figure 7示意裁剪动作块和后退。论文未把这组参数写成全部平台共用。

反应式恢复与离线反思

• 触发：Visual Judge把当前图像与子任务开始时的场景比较，报告running、success或failed。success推进计划；failed则暂停VLA，对同一子任务调用恢复。技能库最初是所有目标共用的一个基础技能；有已接受修订时用当前版本，否则用基础版。
• 基础技能：对掉落物体，先到观察构型再定位目标，生成抓取候选，抓取后抬起，用实测夹爪宽度检查是否握住；通过则运到目的地附近的待命位并做第二次持有检查，通过后由VLA从该位姿继续原指令。检索失败或第二次检查不通过，则标为恢复失败，并尝试回到预抓取位或失败前位姿后张开夹爪。Judge对任务完成的判定独立于恢复是否成功。附录Algorithm 2给出该流程。
• 跨尝试修订：回合之间，反思器查看选定的未解决失败，输入包括场景与目标图像、抓取阶段测量和所用版本。参数记忆P改数值设置，例如接触工具中心点高度；策略记忆S改控制流，例如增加重试或VLM辅助再接地。准入检查拒绝不支持的格式和越界数值。接受的P/S写入下一库版本。VLA权重保持固定。附录D.2说明记录如何使用；两轮协议和失败选择在Section 4.3。

运行时角色

• 模型分工：Table 8写明，规划器和Visual Judge为Qwen3-VL-32B-Instruct，反思器为Qwen3-VL-235B-A22B-Instruct，VLA为本地运行的StarVLA Qwen3-VL-OFT。附录F只定义各模块的输入输出并给出完整提示词。
• 一次请求：无效计划退回规划器。无依赖的子任务可开始，有依赖的按序开始。观察揭示新场景内容后重新规划，下一计划必须用update_scene记录。请求结束时summarizer更新用户记忆（Appendix A）。

Experimental Setup

• Real robot: Franka Research 3, external camera and wrist camera (Figure 5). All real-robot experiments use task-specific fine-tuning of Qwen3-VL-4B-OFT. The planner and Visual Judge are Qwen3-VL-32B-Instruct, and the reflector is Qwen3-VL-235B-A22B-Instruct (Table 8).
• Planning: Sandwich compares Textual, Memory, Preference update on four ingredient layouts. Packing compares Memory, No-habit, Isolation. Lid ingredient compares Write and retrieve, Search without memory, Revise stale memory. The metric is the proportion of dispatched subtasks judged to meet task requirements and applicable preferences. RoboLab has four endpoints: garage, dining table, kitchen, study desk, with 88 requests per text or image condition; Table 2 notes no physical execution.
• Safety: LIBERO-Safety includes TSA and FSHOA, each with 450 episodes; Table 3 has 150 per suite–level cell. Policies are VLAct, OpenVLA-OFT, π0.5, π0.5-SFT, each compared with no additional layer, adapted VLSA, and this paper's shield. Metrics are task success rate without violations (VF) and the proportion of episodes with at least one constraint violation (Viol.). The official SFT checkpoint is released by reference [36], trained on 19,664 safety demonstrations. The paper does not define the difficulty meanings of L0/L1/L2.
• Recovery: 200 episodes on LIBERO-Object with a 12 cm positive x drop perturbation. R0 is the base skill with P/S empty; after each round, at most three failure episodes with recovery evidence per object are used for reflection. Baseline and reflection use Qwen3-VL-235B-A22B-Instruct. The Harness VLA uses the same Qwen3-VL-4B-OFT checkpoint as this paper; CaP-X does not require a VLA. LIBERO-Recovery consists of 77 Object scenes, 5 trials each with closed and open grippers, totaling 770 cases, with the Judge disabled.
• Real-robot execution: Obstacle avoidance has 20 episodes each; recovery after object removal uses five positions, two orientations, two repetitions, 20 episodes per condition. The integrated demonstration serves two food types, carrot and meat; the paper does not report quantitative success rates.
• Evaluation: Execution progress is given by the Visual Judge (Qwen3-VL-32B-Instruct) as running, success, or failed. The judge for dispatch correctness is not specified as another model or human.

Main Results

From the Overall column of Table 3 (VF / Viol.). +VLSA is not listed in the table below.

Policy	No shield	+Ours
π0.5	6.7% / 71.1%	14.6% / 22.7%
π0.5-SFT	25.3% / 52.6%	25.6% / 6.1%
OpenVLA-OFT	6.9% / 61.4%	14.8% / 17.2%
VLAct	4.3% / 69.3%	13.9% / 26.1%

• The authors state that after safety fine-tuning, constraint violations still occur without a shield; this paper's shield reduces violations, while π0.5-SFT's VF is close to that without a shield. The adapted VLSA's Overall violations are: π0.5-SFT 37.7%, π0.5 49.8%, OpenVLA-OFT 42.0%, VLAct 44.8% (Table 3).
• The authors state that the same shield reduces violations across these backbones. Not all difficulty-level cells are close to Overall: π0.5+Ours FSHOA L2 is 2.0% / 38.7%, VLAct+Ours FSHOA L2 is 1.3% / 37.3%.
• On the real robot, a carrot-carrying policy trained without obstacle data goes from 0/20 to 20/20 contact-free completions after inserting obstacles (Table 1). The authors state this is repeated intervention against collision risk.

Personalized Planning and Memory Revision

• Real-robot dispatch (Table 1): Sandwich Textual and Memory are both 24/24, after preference update 20/24 (83.3%). Packing Memory, No-habit, Isolation are 16/16, 16/20, 16/18. Lid ingredient three conditions are 12/12, 8/10, 9/12. The authors state the system retains preferences across dialogues and follows revised preferences on 20/24 subtasks; recorded failures include misidentification and missed subtasks.
• RoboLab (Table 2): Without memory, text and image correct selections are both 77/88, median dispatch times 28.6 s and 34.6 s; with memory, both 88/88, times 7.0 s and 7.1 s. After moving the target, memory correction yields text 76/88, image 77/88, and subsequent dispatch 83/88 and 85/88. Correction requires writing the new location and deleting the outdated association. The authors state both metrics support observation-driven revision and subsequent reuse.
• Authors' interpretation: These studies cover requirement acquisition, retention across interactions, and revision after changes. The authors state the results support connecting memory to planning and observation, rather than treating stored locations as permanently valid instructions.

Recovery on Perturbations and Recorded Failures

• 12 cm perturbation, 200 episodes (Table 4): The tested VLA is Qwen3-VL-OFT itself. Without recovery, 26 completions (13.0%); CaP-X 10 (5.0%), Harness VLA 101 (50.5%). CarryOn R0 58 (29.0%), R1 cumulative 141 (70.5%), R2 cumulative 160 (80.0%). R1 solves 83 of the previous 142 failures, R2 solves 19 of 59. The abstract and introduction write from 13% to 80%; Table 4 shows 13.0% and 80.0%. R0 is lower than Harness VLA; the authors state the full iterative system has the highest completion rate.
• LIBERO-Recovery, 770 cases (Table 5): Judge disabled, recovery directly after initialization. Pure VLA 247 (32.08%), R0 385 (50.00%), R1 569 (73.90%), R2 586 (76.10%). R1 solves 184 of 385, R2 solves 17 of 201. The authors state the base recovery improves by 17.92 percentage points under the same rollout budget.
• Real robot: After object removal, RAW 3/20, BASE (z 0.00, 1 attempt) 9/20, FULL (z −0.02, up to 3 attempts with perception update) 18/20 (Table 1). The authors state external repair is higher than the original policy, and retained revisions further improve completion.

Real-Robot Integrated Demonstration

• What was tested: A vegetarian preference is stored; the user asks to serve lunch from a covered pot; the system must open the lid, judge whether it can serve, and bypass obstacles when the operator interferes with the carry. In the second trial, the lid covers meat, while scene memory still records carrot. The policy is trained on obstacle-free data for lid opening and food retrieval (Section 4.4).
• Process described by the authors: In the carrot trial, scene memory guides lid opening, the shield avoids the rice cooker, after a drop the Judge triggers re-grasping, then returns to the VLA to place in the bowl. In the meat trial, observation contradicts memory; the system replaces the stale carrot entry with meat and refuses to serve because it does not meet dietary requirements (Figure 6).
• Authors' interpretation: The authors state the dual-loop role is to distinguish between "repairing an interrupted but still appropriate task" and "stopping when new evidence makes the task no longer appropriate." The paper does not report success rates for these two demonstrations.

Other Ablations and Analyses

• Average single-call times: geometric filtering 21.06 ms, VLA inference 47.82 ms, perception 170.82 ms; the authors state perception is the main component (Section 4.2).
• Table 4 cumulative API tokens per original episode: R2 41,109, of which reflection 2,609; CaP-X 1,124,819, Harness VLA 3,829,605. The authors state higher completion with lower recorded VLM tokens.
• Table 5 text states final cumulative 7,158.77 tokens/case; table R2 is 7,159.
• Appendix D.3 BBQ sauce chain: R0 3/20, selected failure contact height 66.0 mm, descent stuck at 109.0–113.2 mm; P first changed to 110.0 mm, R1 3/17, then changed to 105.0 mm, R2 11/14, cumulative from 3 to 17/20.
• Policy-only probe: after 12 cm displacement of the object relative to initial lift position, frozen StarVLA completes 26/200 (13.0%) (Section 3.3, Figure 3). The caption states blue bars are task success, and the upper line is the percentage-point drop relative to no perturbation; the text does not give bar heights per displacement.
• Dispatch is not always perfect: preference update 20/24, stale scene memory revision 9/12, no-habit packing 16/20 (Table 1).

Limitations and Future Directions Noted by the Authors

The paper has no separate Limitations section. The Conclusion explicitly states future directions:

• Longer-term adaptation: The authors write they will evaluate longer-term adaptation in the future (Conclusion).
• Cross-task transfer: The authors write they will evaluate transfer of recovery experience across tasks in the future (Conclusion).
• Protection and completion: The authors write they will improve the balance between protective intervention and task completion in the future (Conclusion).

Experimental Coverage

• Real robot is Franka Research 3. Planning includes three sandwich conditions, three activity-related packing conditions, three lid ingredient conditions; execution includes obstacle insertion and recovery after object removal; plus two food service demonstrations (Section 4.1–4.4, Table 1).
• LIBERO-Safety includes TSA and FSHOA, each with 450 episodes; Table 3 has 150 per suite–level cell. Comparisons include π0.5, π0.5-SFT, OpenVLA-OFT, VLAct, as well as adapted VLSA and this paper's shield.
• Recovery is 200 episodes on LIBERO-Object with 12 cm positive x drop, and LIBERO-Recovery with 77 Object scenes, 770 cases. Reflection is at most two rounds, with at most three failure episodes with recovery evidence per object (Section 4.3, Table 4, Table 5).
• RoboLab has four endpoints, 88 requests per text or image condition; Table 2 is planning without physical execution. The target-moving experiment has an additional 88 pairs of requests.
• Planner and Visual Judge use Qwen3-VL-32B-Instruct, reflector uses Qwen3-VL-235B-A22B-Instruct (Table 8). The paper does not report the judge for dispatch correctness, agreement with humans, number of repetitions, or statistical tests.

CarryOn is a dual-loop harness wrapped around a frozen VLA for personalized robot assistance with execution-time protection and repair.

• Problem: The authors argue that following instructions alone is insufficient to serve daily requests. The system must decide subtasks according to changing user preferences and scenes, intervene before dangerous contact, repair after physical failure, and leave failure experience behind.
• Method: The planning loop uses user profiles and location-based scene memory together with the current request and observations to dispatch subtasks with endpoints; dialogue and observations then revise memory, and missed required scene updates are returned. The execution loop performs geometric distance filtering before the frozen VLA and calls control recovery skills after failure. Offline reflection revises only recovery parameters and policies and retains versions; the VLA and shield are unchanged.
• Planning results: On the real robot, memory-conditioned dispatch is 24/24 when only saying to make a sandwich, and 20/24 after preference updates; packing is 16/16 with a profile. On RoboLab without physical execution, with memory, correct text and image selections are both 88/88, with median times of 7.0 s and 7.1 s respectively (Table 1, Table 2).
• Safety results: On LIBERO-Safety Overall, π0.5-SFT's violation rate drops from 52.6% to 6.1% with the shield, and VF from 25.3% to 25.6%. Original π0.5's violation rate drops from 71.1% to 22.7%. Violations also drop on OpenVLA-OFT and VLAct, but cells such as FSHOA L2 remain high, e.g., π0.5+Ours at 38.7% (Table 3). On the real robot with added obstacles, contact-free completion goes from 0/20 to 20/20.
• Recovery results: Over 200 12 cm perturbation episodes, the same VLA without recovery is 13.0%, R0 is 29.0%, and cumulative after two reflection rounds is 80.0%; R0 is lower than Harness VLA's 50.5% (Table 4). Over 770 recorded failures, R2 cumulative is 76.10% (Table 5). Real-robot FULL condition is 18/20.
• Authors' conclusion: The authors claim persistent context and physical feedback extend assistance beyond single task execution. Future work will evaluate longer-term adaptation and cross-task transfer of recovery experience, and improve the balance between protection and task completion.