Is a robot's memory just a clock? Temporal shortcuts in memory-augmented VLA policies

Core question: can VLA policies with time-filled memory distinguish "waiting" from "working"—the authors' answer is no; the robot treats idle time as task progress.

• Setting and importance: many manipulation tasks (e.g., "pick up and put down the block K times, then press the button") cannot be solved from the current observation alone, and memory-augmented VLA policies condition each action on episode memory. One simple and widely used approach (per the authors) is time-filling: uniformly sample past frames and stamp them with time steps, and the strongest released policy on RoboMME uses exactly this design.
• Shortcomings of existing methods: real robots often wait because of slow planners, operator pauses, or safety monitoring, and also hesitate or grasp air (the authors call such intervals null events). Time-filled memory assigns sampled frames and timestamps to such intervals and cannot skip them the way memory that "records what happened" would.
• Core observation: demonstrations proceed at a steady pace, so elapsed time alone predicts progress—reaching 89% and 80% prediction accuracy on PickXtimes and SwingXtimes demonstrations, respectively (Fig. 3). Imitation learning never penalizes a policy that reads progress from time, from which the authors propose the temporal shortcut hypothesis.
• What the paper proposes: a diagnostic protocol (visible/hidden null-event contrast + timestamp scaling) to determine whether a memory policy tracks events or time; a causal analysis of the failure mechanism (two paths: timestamps and slot occupancy); and a comparison of two fixes derived from the diagnosis (pause data augmentation vs test-time event indexing).

Memory in robot policies

• Most VLAs are conditioned only on the current observation or a short window; memory directions include history-aware transformers, perceptual memory banks, keyframe/event selection, recurrent and test-time-training memory, and video-language hybrid memory. RoboMME [11] standardized evaluation and released the MME-VLA family studied in this paper; MIKASA-Robo [8] and MemoryBench [13] are related suites.
• A few works ask what memory actually does: history is encoded but barely used [24], and long, unfiltered history introduces spurious correlations [30, 47, 52]. The paper notes that these works treat the problem as history distribution shift and solve it by frame selection, whereas this paper identifies the specific correlate "elapsed time" and verifies it with interventions.
• Closest to this paper: AGM [16] shows that recording attempted actions as progress turns failed grasps into persistent errors; RoboMME-Interference [39] shows that irrelevant sessions crowd out frame-sampled memory; concurrent EventVLA [53] and KEMO [57] report that event-based frame selection outperforms uniform sampling.

Shortcuts, time, and position

• Shortcut learning [17] has documented causal confusion [12] and copycat behavior [10, 50] in imitation learning; time is a legitimate state in time-limited tasks [34], many controllers are driven by phase variables [20, 35], and GVL [29] therefore shuffles frame order.
• In sequence models, positional encoding affects counting and length generalization [1, 21, 37, 40]; video moment retrieval models can answer from the temporal position of the moment rather than its content [33, 55].

Events and redundancy in video memory

• Video models compress long inputs by merging redundant frames/tokens [5, 19, 43], and event segmentation groups frames into meaningful units [15, 42, 56]; the paper notes that these methods judge redundancy by appearance, whereas null events are not necessarily visually redundant (a jittering wrist changes every frame). This paper's event indexing instead uses "whether the world has returned to a previous state", similar to SLAM loop closure [7].

Robustness of robot policies

• Existing benchmarks perturb objects, textures, lighting, cameras, language, and initial states [14, 38, 49, 59]; on the time dimension only inference latency has been studied [4, 26]. The authors state that to their knowledge, no prior work injects test-time null events into rollouts of memory-augmented policies.

Positioning: the authors position this paper as a causal dissection of "what memory actually does", and event indexing as a minimal intervention to test that explanation rather than a new architecture.

Overall approach: inject null events into counting tasks, use a visible/hidden contrast to separate "physical interruption" from "memory content", then localize the failure mechanism using timestamp scaling, a model without time encoding, and cross-design comparison, and finally test two fixes suggested by the diagnosis.

Tasks, policy, and null events

• Tasks: two counting tasks in RoboMME—PickXtimes (pick up and put down the block K∈{1..5} times, then press the button) and SwingXtimes (carry back and forth K times), where memory is used to decide when to stop.
• Policy under test: the released MME-VLA (based on π0.5), whose FrameSamp memory keeps 32 frames sampled uniformly over time, each frame with a sinusoidal encoding of the absolute time step, read through different paths by three variants (Modul/Expert/Context); TokenDrop selects tokens by pixel change. Unless stated otherwise, results use the strongest FrameSamp-Modul.
• Three types of null events: injected at action-chunk boundaries after policy steps 48 and 160—pause (hold still for 100 environment steps = 5 seconds; dose study uses 30 and 300 steps), wrist jitter (roll ±0.3 rad around the target for 100 steps), air grasp (the planner executes a grasp above a stationary object and returns to the original pose). The evaluator records no object movement exceeding 1 cm.
• Visible vs hidden: when visible, null-event frames enter memory normally and timestamps advance as usual; when hidden, the physical event is exactly the same, but the frames are not written to memory and timestamps do not advance. The hidden condition is the reference for "memory that is indifferent to null events".

Protocol and statistics

• 50 test episodes per cell; all comparisons pair the same episodes (environment seed and policy noise), using the exact two-sided McNemar test; failures are classified as early (pressing the button/submitting before completing K times) or extra (doing more than K times).
• Evaluation is not fully deterministic: two runs of the same condition differ by 8/50 (Sec. E), so differences within about 10 episodes are interpreted cautiously.

Four steps of the mechanism analysis

1. Time predicts progress (Sec. 5.1): at each demonstration step, predict the number of completed repetitions from elapsed time alone (10-step bins, 5-fold cross-validation), against a "always guess the most common count" baseline.
2. Scale timestamps (Sec. 5.2): inject no null events and leave frames unchanged, only multiply all memory-frame timestamps by a factor s; if the policy reads progress from time, a fast clock should cause early stopping and a slow clock should cause extra repetitions.
3. Remove time encoding (Sec. 5.3): train from the π0.5 base per the released recipe for 30k steps a Clock (keeps time encoding) and a NoClock (removes it), differing only in the use_pos_emb line; together with three controls: null swap (replace null frames with the last frame before the pause), duplicate fill (fill slots with copies of real frames), null block (insert an identical frame block without a pause).
4. Cross-design comparison (Sec. 5.4): apply the same null events to all released memory designs in RoboMME (including MemER-style keyframes, textual subgoals, TTT, and memoryless π0.5) and to two independent policies SimpleMemVLA and LTM-π, without retraining.

Two fixes

• Change the data: inject 1–3 synthetic pauses of 20–600 steps into the memory timeline of 80% of training samples (repeat one frame, timestamps advance as usual), plus a weaker dose (30% of samples, at most 2 segments of 20–100 steps).
• Change the index (event indexing): no retraining at test time; if the world returns to a previous state (robot configuration returns within the tolerance of some retained frame, the scene exterior is unchanged, and no contact change occurred in between), that segment is collapsed into one frame, and long stationary periods are likewise collapsed, with temporal positions recomputed from the retained frames; the detector reads the robot silhouette, gripper width, and external force on the arm, with tolerances fixed across all 16 RoboMME tasks (Sec. I). The authors state that this experiment tests "whether removing null frames from the index is sufficient", not that it provides a deployable detector.

Experimental setup

• Models tested: released FrameSamp-Modul/Expert/Context, TokenDrop (three variants), MemER-style keyframe, text subgoal, TTT, memoryless π0.5; independent policies SimpleMemVLA, LTM-π; Clock and NoClock trained from scratch (π0.5 base, 30k steps).
• Benchmarks: RoboMME's PickXtimes and SwingXtimes (main analysis), all 16 tasks for generalization (Table 9); MIKASA-Robo's BlinkCount/TimedTransfer as exploratory studies.
• Metrics: success rate (number of successes out of 50 paired episodes); failure classification early/extra; exact two-sided McNemar test with Holm correction over a family of 42 comparisons (Sec. L).
• Protocol: 50 episodes per cell, paired episode by episode; held-out confirmation using 50 validation episodes per task; key tests pre-registered (Sec. K).
• Repeats and seeds: Clock/NoClock were trained with two seeds, 42 and 43; the remaining models (including each augmentation dose) have one seed; retraining Clock with the same seed differs by at most 12 episodes on PickXtimes no-event cells (Table 4).

Main results

According to Fig. 2b and Table 1 (FrameSamp-Modul, two 5-second events, 50 paired episodes):

Swipe horizontally to read the table

Task	No event	Pause (visible)	Jitter (visible)	Air grasp	Pause (hidden)
PickXtimes	47	16	16	not used in main text	45
SwingXtimes	44	10	10	23	47

• Failure form: under pauses and jitter almost all failures are early stopping (PickXtimes 34/34, SwingXtimes 36/40, Sec. B); episodes with K=1 almost all survive, and none with K≥2 succeed under pauses/jitter (Table 1).
• Causally attributed to memory: the hidden condition keeps physical events unchanged and only omits them from memory, and success in every cell returns to the no-event level (Fig. 2b); the authors conclude on this basis that the damage comes from what the policy remembers.
• Dose and timing: damage appears between 0.5–1.5 seconds per pause and saturates at 5 seconds (Table 6); a single pause only harms episodes that still require a stop decision afterward.

Timestamp scaling (Sec. 5.2)

• What was measured: no null events, frames unchanged, only the timestamps of memory frames multiplied by s (Fig. 4, FrameSamp-Modul, PickXtimes).
• Results: at s=1.5 success drops from 47 to 18, with all failures being early button presses; at s=0.5 it drops to 24, with failures turning into extra repetitions (23 extra/3 early); ×0.5 stays within the training distribution. FrameSamp-Expert flips in the same direction (Table 13).
• Authors' interpretation: the direction of failure follows the direction of the clock, and a null event is equivalent to a local acceleration of the clock; under a 15-second long pause, 34/36 of Clock's early presses occur after at most 1 repetition, and 29/29 on held-out (Fig. 5), which the authors say shows the policy reads the duration of the null event.

Removing time encoding and cross-design comparison (Sec. 5.3–5.4)

• NoClock still fails: NoClock without time encoding is unaffected by ×1.5 (44 vs 42) and counts comparably, but two 5-second pauses still reduce it to 21 (PickXtimes) and 14 (SwingXtimes), with recovery after hiding (Fig. 6a); null swap (replacing null frames with the last frame before the pause) leaves the damage unchanged, and duplicate fill (filling slots with copies of real frames) does not harm.
• Empty-slot dose: inserting identical frame blocks with no pause, 12 empty slots reduce NoClock from 42 to 24 (PickXtimes) and from 47 to 23 (SwingXtimes) (Fig. 6c/Table 23).
• Across designs (Table 2, two null events): all three FrameSamp variants collapse (e.g., Modul 47→16); TokenDrop selects tokens by pixel change, so pause damage is smaller (39→30) but jitter damage is larger (39→21); subgoal-indexed MemER-style (38→39) and text subgoal (46→48) are unaffected by either event on PickXtimes; SimpleMemVLA, which writes at every step, is almost unharmed under pauses on PickXtimes (50→47) while LTM-π collapses (50→7).

Two fixes (Sec. 6)

• Pause augmentation fails: trained on a second machine and compared with same-machine retraining (Fig. 8a/Table 28): pauses no longer cause damage on PickXtimes only because no-event success has already dropped from 32 (retraining) to 24; on SwingXtimes performance is comparable to retraining without augmentation (10 vs 11); on held-out, K≥2 drops from 35 to 7/40. A weaker dose preserves SwingXtimes counting but gives no improvement under pauses.
• Event indexing works: enabled at test time on the same Clock checkpoint (Fig. 8b/Table 30): success at least doubles under all three null events, and SwingXtimes rises from 13 to 43 under air grasping; reproduced on held-out and on released policies (Table 34, 6 of 7 probes p<.05).
• Boundaries: event indexing has a cost on PickXtimes with no events (released policy held-out 139→120/150, pre-registered confirmation); recovery is limited for one-way air grasps that do not return to the original state and for random micro-pauses (Table 38: 20→27, 17→24, not significant); on TimedTransfer, where time itself is the task, collapsing pauses is wrong by construction.

Other ablations and analyses

• 16-task generalization (Table 9): video-conditioned tasks are harmed through another mechanism—null frames push the demonstration video out of the 32 slots, and RouteStick drops from 33 to 7, reproduced and recovered by starvation/protection controls.
• Time predicts progress in demonstrations (Table 10): PickXtimes 0.890, SwingXtimes 0.802; real teleoperation three-cycle data 0.922–0.933, comparable to simulation.
• Linear probe (Table 22): time is still linearly readable from NoClock memory tokens (held-out R² 0.92/0.89), but the pause-inflation criterion is met only on PickXtimes, and the authors draw no conclusion per the pre-registered rules.
• MIKASA-Robo exploration (Sec. J): BlinkCount base success is only 24/50 and the pre-registered threshold is not met; with a fixed cue duration of 50 steps success drops to 6 and concentrates at K=5 (5/9), which the authors say is consistent with "reading K from cue duration", with exploratory evidence.
• Negative results: PickXtimes air grasping is excluded from the main text because the returning gripper often regrasps the block rather than being a null event (Sec. B); the predicted pause damage for SimpleMemVLA is not met (47 vs 50, Sec. K).

Limitations and future directions noted by the authors

• Limited coverage (Sec. 7 Limitations): the causal analysis covers only one family of memory policies (MME-VLA) and models retrained with its architecture; the two independent policies were only probe-tested and not dissected.
• All in simulation (Sec. 7): all experiments are in simulation, every null event is script-injected, and the frequency of unannounced null events in deployment was not measured; the event detector also uses simulated signals.
• Seeds and training variance (Sec. 7): Clock and NoClock have a second seed and satisfy all pre-registered rules, but the remaining models (including each augmentation dose) have only one seed; Clock retraining differs by at most 12 episodes on PickXtimes no-event cells, so separately trained models are compared only when the gap is larger.
• Open question (Sec. 7): without timestamps, which feature the policy reads from an occupied slot remains unknown.
• Future directions (Sec. 7 Discussion): detecting world changes from real sensors is harder than in simulation, which the authors regard as the main open problem for "memory that tracks what happened"; null events, clock scaling and fixed-duration cues are cheap probes any memory benchmark can add.

Experimental coverage

• 14 models were tested: three FrameSamp variants and three TokenDrop variants of MME-VLA, MemER-style keyframe, text subgoal, TTT, π0.5, the independent SimpleMemVLA and LTM-π, and Clock and NoClock trained from scratch (Sec. 3, Sec. G).
• Tasks cover all 16 RoboMME tasks (Table 9, of which 2 are at floor and not run, and 4 are reported but not interpreted due to fewer than 15 no-event successes), plus MIKASA-Robo's BlinkCount and TimedTransfer (exploratory, Sec. J).
• 50 paired test episodes per cell, with held-out confirmation using 50 validation episodes per task; evaluation is not fully deterministic, with two runs of the same condition differing by 8/50 (Sec. 3, Sec. E).
• All causal evidence comes from simulation; real teleoperation data (austin_sailor, stanford_hydra, FMB, LIBERO-10) is used only for correlation statistics on "time predicts progress", and the authors explicitly state this does not show that policies trained on real data would use this shortcut (Sec. D).
• The event detector uses simulated segmentation and noise-free rendering, gripper width and external force are simulation stand-ins, and the 5-pixel scene tolerance needs recalibration on real cameras (Sec. I.2, Sec. I.5).

A causal analysis paper on memory-augmented VLA policies that proposes and dissects the phantom progress failure: robots whose memory is filled by time treat idle time as task progress.

• Problem: on RoboMME counting tasks, two 5-second pauses or wrist jitter reduce the released FrameSamp-Modul from 47/50 to 16/50 on PickXtimes and from 44/50 to 10/50 on SwingXtimes, with failures almost all being early button presses; after hiding the event frames (physical events unchanged, not written to memory) success recovers, and the authors conclude on this basis that the cause lies in memory rather than in the interruption itself.
• Mechanism: demonstration pacing is stable, so elapsed time alone predicts progress (PickXtimes 89%, real teleoperation data over 92%), and imitation learning therefore learns a temporal shortcut. Scaling timestamps alone (no null events) can produce early or repeated presses in the corresponding direction; but NoClock, which removes time encoding, still fails (21 and 14/50), because null event frames still occupy time-sampled memory slots—the shortcut lies in the indexing method, and timestamps are only one carrier.
• Across designs: the authors state that fragility follows the indexing method—time-indexed memories all collapse, pixel-change-selected TokenDrop is affected more by jitter than by pauses, and subgoal-indexed memory is unaffected by either null event on PickXtimes (Table 2).
• Fixes: pause data augmentation does not eliminate the failure and harms counting when there are no events; test-time event indexing (collapsing segments where the world returns to a previous state into one frame) at least doubles success under all three null events without retraining, raising SwingXtimes from 13 to 43/50 under air grasping, but it has a cost on PickXtimes where genuine repetitions also form loops, and recovery is limited for micro-pauses and one-way events.
• Authors' conclusion: memory intended to record "what happened" should advance with events rather than with the clock; time should be kept as a policy input only when duration itself is the task. The authors believe the shortcut is unlikely to be specific to RoboMME and suggest that memory benchmarks add null-event and clock-scaling probes.