From Perception to Prediction and Intervention: Evaluating Proactive Video Assistants in Real-World Scenarios

Existing video benchmarks mainly check whether a model can explain what it has already seen, while real streaming assistants also need to decide whether and when to help when there is no user query.

• Use case: The authors say streaming video lets an assistant continuously observe a changing environment, and proactive understanding requires the model to autonomously decide whether to reply and when. Replying too late is useless, and triggering without sufficient evidence lacks justification.
• Existing gaps: The authors say existing proactive video benchmarks mainly do perception, such as recognizing ongoing events or predefined conditions. Real assistance also requires early warning of risks that have not yet occurred, and giving reminders, corrections, or guidance according to user goals. Prediction and intervention demand stricter precision in labeling the response time.
• Authors' breakdown: Proactive assistance requires simultaneously handling what is currently visible, what may happen next, and when guidance becomes necessary. The authors map these three to perception, prediction, and intervention, respectively.
• This paper proposes: The authors propose P3IBench (Proactive Perception-Prediction-Intervention), containing 533 video-queries and 429 non-duplicate videos; they use it to evaluate 12 open-source and proprietary video language models; and they additionally build proactive training data, training P3I-Assistant-4B on Qwen3-VL-4B.
• Evaluation setup: The evaluated object is a causal streaming video assistant. The input is a task instruction and frames up to the current time, and the output is either remaining silent or giving a reply at a certain video moment. Scoring considers both whether the content is close to the label and whether the emission time is close to the labeled time.

Online and Proactive Video Models

• VideoLLM-online, MMDuet, MMDuet2: Chen et al. (2024) use causally interleaved video–language interfaces for temporally aligned streaming dialogue; Wang et al. (2025c) use video–text duet for time-sensitive interaction; Wang et al. (2026)'s MMDuet2 uses multi-turn reinforcement learning to learn to respond or remain silent.
• VideoChat-Online, Dispider, Em-Garde: Huang et al. (2025) use Pyramid Memory Bank to maintain online context; Qian et al. (2025) separate perception, decision, and reaction; Zheng et al. (2026) use lightweight proposal matching to trigger responses.
• VideoChat3, JoyAI-VL-Interaction: Li et al. (2026) and Yao et al. (2026) support proactive response control in continuous video streams. The paper does not critique these models individually in this section.

Streaming and Proactive Video Benchmarks

• StreamingBench, OVO-Bench, ODV-Bench: Lin et al. (2024) and Niu et al. (2025) evaluate online perception and prediction-biased tasks; Zeng et al. (2025) also cover perception and prediction, with scenarios focused on driving.
• ProactiveVideoQA, RIVER, LiveProBench: Wang et al. (2025b), Shi et al. (2026), and Du et al. (2026) are classified by the authors as perception-centric interactions.
• EgoPro-Bench, OmniPro: Ran et al. (2026) evaluate personalized proactive assistance in egocentric activities; Zhao et al. (2026)'s OmniPro includes Sequential Step Instruction as a procedural guidance subtask. The authors state that previous benchmarks typically cover only a subset of Perception, Prediction, Intervention, or combine them only in specialized settings.

Baseline Methods and Benchmarks

• Baselines: 8 offline MLLMs adapted to causal streaming at inference time, and 4 native proactive streaming models evaluated under their respective inference frameworks. The benchmark is P3IBench; samples come from Ego4D, TAG, Nexar, UCF-Crime, and self-recorded first-person videos in five scenarios. Table 1 also lists the scale, scenarios, capabilities, and annotation methods of StreamingBench, OVO-Bench, ProactiveVideoQA, ODV-Bench, RIVER, EgoPro-Bench, OmniPro, and LiveProBench.

The authors position this paper as: simultaneously evaluating routine monitoring, early risk warning, and goal-conditioned assistance under a unified causal protocol, and using task-specific automatic annotation, targeted verification, and human inspection to provide response targets, thereby controlling whether responses are valid and their time boundaries.

The authors use P3IBench to frame proactive video understanding as stepwise revealed decisions: the model sees only current and previous frames, replies when it should, and otherwise remains silent. The benchmark is divided into Daily Perception, Risk Alert, and Event Intervention, roughly corresponding to perception, prediction, and intervention.

Task setup

• Input and output: A sample is initialized with instruction I. At video time t, the model sees V≤t within the context window and reply history H, and either remains silent or emits oj=(tj, ŷj). Each target gi=(τi, yi) constrains both time and content.
• Daily Perception: Event Monitoring is divided into atomic events, processes composed of multiple time-related events, and semantic categories that can correspond to multiple atomic events; a reply should be given once the target can be reliably identified. Object Monitoring tracks a specified object and outputs Warning when it disappears and can be confirmed; disappearing again after reappearing counts as a new target.
• Risk Alert: Accident Warning requires outputting Warning at the earliest moment before a collision occurs when the risk can be reliably identified, once per video. Crime Alert outputs one Warning when criminal or suspicious behavior becomes detectable.
• Event Intervention: According to the user's goal and observed progress, give reminders, corrections, or guidance within a useful window. A video can have multiple interventions, without waiting for a new user query.

Data construction

• Event Monitoring: From Ego4D, using its timestamped dense narration to generate candidate queries, match events with visual evidence, and locate boundaries. Two independent vision language models estimate boundaries, followed by consistency checks, verification, and confidence-based filtering. Appendix A gives details; the paper does not name these two models.
• Object Monitoring: From TAG, using segmentation masks to determine object disappearance and timestamps, then checking or manually correcting.
• Risk Alert: Accident Warning comes from Nexar driving videos and their original accident annotations, with manual inspection. Crime Alert comes from UCF-Crime, with coarse intervals manually refined.
• Event Intervention: Self-recorded first-person videos in scenarios of grocery shopping, navigation, procedural maintenance, desk organization, and item packing. Each script contains at least one deliberate error in step order or task content. Intervention windows and reference replies are all manually labeled and verified.
• Scale: 533 queries, 429 videos, 836 GT response targets. In Figure 2, the five task types are Event Monitoring 233 (43.7%), Object Monitoring 50 (9.4%), Accident Warning 100 (18.8%), Crime Alert 100 (18.8%), and Event Intervention 50 (9.4%). The proportions of GT falling in the first, middle, and last third of the video are 37.8%, 39.7%, and 22.5%; Event Intervention in these three segments is 87, 87, and 84, respectively.

Evaluation metrics

• Pairwise score: For a matched reply and target, s(o_j,g_i)=c(ŷ_j,y_i)exp(−α|t_j−τ_i|), α=0.05. c is content correctness: structured outputs get 0 or 1 from deterministic rules; open replies in the intervention subset get [0,1] from an LLM. The paper does not name this review model. The exponential term decays with the absolute deviation between emission time and target time.
• Redundancy penalty: Each reply matches at most one target. Target score S_i=max_{oj∈ Oi}s(o_j,g_i)/|O_i|, and 0 if no match. The sample score is the average of target scores.
• Two matching modes: PRS-U assigns a reply to the nearest target before its emission time; replies before the first target are assigned to that target but get a score of 0, still counting toward the match count. PRS-B assigns to the temporally nearest target, so early replies can match future targets. The correctness rules and aggregation rules are the same for both.

Inference adaptation and post-training

• Offline model adaptation: The system prompt specifies the role, task, and when to remain silent; videos are processed at 1-second intervals, with new frames appended at a default 1 FPS, and the model decides at each step whether to reply. The context retains the initial instruction and previous inputs and outputs. Most models retain all observed frames; GPT-5.6-Luna and InternVL3.5-8B retain only the most recent 16 and 32 frames, respectively, with full text history retained. Appendix B gives the procedure. The silence token for most models is Silence; for JoyAI-VL-Interaction it is </silence>, and for MMDuet2 it is NO REPLY.
• Native models: They follow their own inference frameworks, with P3IBench task requirements written into the prompt.
• P3I-Assistant-4B: SFT on Qwen3-VL-4B-Instruct followed by GRPO. SFT freezes the vision encoder and trains the language model along with the vision merger and alignment modules; focal and frequency-balanced weights are used for </Silence> and </Response>. Appendix C does not give the focal exponent γ value. SFT learning rate is 1×10−5, 1 epoch, 418 steps. RL starts from the step-418 checkpoint, learning rate 5×10−7, about 1 epoch, 143 steps, 4 generations per input, KL coefficient 0. Accident and Compositional add extra penalties for redundant or invalid outputs, λ=0.2. The authors say dynamic weighting is added to mitigate the over-silence observed in Section 4.2.

Experimental Setup

• Models under test: Adapted offline models are GLM-4.1V-9B-Base, InternVL3.5-8B, Qwen2.5-VL-3B-Instruct, Qwen3-VL-4B-Instruct, Qwen3-VL-8B-Instruct, Qwen3.5-4B, Qwen3.5-9B, GPT-5.6-Luna. Native proactive models are JoyAI-VL-Interaction (written as JoyVL in tables), MMDuet2, Em-Garde, VideoChat3. The post-trained model is P3I-Assistant-4B.
• Data: The evaluation set is P3IBench, with 533 queries, 429 videos, 836 GT targets; sample sizes for the five task types are shown in Figure 2. Training additionally uses 10,000 PVQ from Shot2Story, 2,000 CRR from LLaVA-Video, and 600 Accident and 768 Event samples.
• Metrics: Main metrics are PRS-U and PRS-B, with α=0.05. PRS-U′ and PRS-B′ in Appendix E remove the division by the number of matched responses, taking only the highest match score per target. Figure 3 shows the proportion of GT windows with 0, 1, 2, ≥3 responses; Figure 4 shows first-response latency, excluding windows with no response.
• Inference: Default 1 FPS, 1 frame per step. GPT-5.6-Luna retains the last 16 frames, InternVL3.5-8B retains the last 32 frames. Structured outputs are judged by rules; open-ended intervention responses are scored by an LLM, but the paper does not specify the reviewer model. The paper does not report the number of repetitions.

Main Results

The table below shows the average scores and PRS-B for some tasks from Table 2. Since it exceeds 8 rows, we retain the highest, lowest, and several intermediate average PRS-B models. InternVL3.5-8B (average PRS-B 26.79) is omitted; Em-Garde is already included; also omitted are Qwen3.5-4B (26.16), Qwen3.5-9B (26.11), Qwen2.5-VL-3B-Instruct (16.70). P3I-Assistant-4B is not in Table 2.

Swipe horizontally to read the table

Model	Average PRS-B	Average PRS-U	Accident PRS-B	Intervention PRS-B	Event PRS-B
GPT-5.6-Luna	46.59	38.78	74.64	29.85	24.60
Qwen3-VL-8B-Instruct	41.73	27.79	68.83	5.53	26.00
JoyVL	31.62	21.36	50.62	4.10	14.78
Qwen3-VL-4B-Instruct	28.62	24.57	41.78	9.13	21.35
VideoChat3	28.11	23.13	53.38	18.77	12.91
InternVL3.5-8B	26.79	21.27	38.79	5.89	9.06
GLM-4.1V-9B-Base	10.20	9.62	2.80	0.93	0.96
MMDuet2	8.86	8.84	0.00	1.03	0.26

• The authors state that no model is reliable across all three capabilities; the strongest system's average PRS-B is only 46.59 in Figure 1b; most models have average PRS below 30. The authors state that Intervention is particularly difficult.
• The authors state that GPT-5.6-Luna is the strongest overall and has advantages over open-source models in risk prediction and intervention, but still struggles to perceive emerging events and objects in a timely manner. In Table 2, its Event and Object PRS-B are 24.60 and 41.51, lower than Qwen3-VL-8B-Instruct's 26.00 and 50.33.
• The authors state that adapted offline models tend to produce continuous outputs, causing redundancy; native proactive models, after reducing redundancy, often remain silent in intervals where they should respond.

Response Count and Latency

• What was measured: Figure 3 counts the number of responses within each GT window; Figure 4 shows the median and interquartile range of first-response latency.
• Results: The authors report that Qwen2.5-VL-3B-Instruct gives at least 3 responses in 53/100 GT windows for Accident Warning, with PRS-B of 21.27; if the redundancy penalty is removed, it is 71.39. The corresponding PRS-B′ in Table 8 is 71.39. The authors state that MMDuet2 and JoyVL miss 79% and 100% of GT windows on this task, respectively; in Table 2, MMDuet2's Accident scores are both 0.00, and JoyVL's are 19.66 and 50.62. In Figure 4, VideoChat3's median latency for Event and Crime is 51.4s and 31.4s; MMDuet2's Object and Accident are marked as No response, with Crime median at 23.8s.
• Author interpretation: The authors state that object perception is relatively timely, while event-level perception requiring continuous context is weaker, further affecting the timeliness of prediction and intervention. The authors state that PRS-B is generally 6–15 points higher than PRS-U, with a particularly large gap on Accident Warning; the overall gap in Table 2 does not always fall in this range, e.g., GLM-4.1V-9B-Base is 9.62 vs 10.20, MMDuet2 is 8.84 vs 8.86. The authors speculate that excessive silence may stem from an imbalance between silence supervision and content-response supervision, citing Xia et al. (2026).

Streaming Context

• Duration and event timing: Figure 5 bins videos by duration and GT event time, comparing Qwen3-VL-4B-Instruct and JoyVL. The authors state that longer videos and later events are harder, and that Qwen3-VL is less sensitive to information at the beginning of videos. In the figure, Qwen's median PRS-U for non-zero responses across duration bins is 0.45, 0.44, 0.19, 0.17, 0.08; JoyVL's is 0.49, 0.75, 0.85, with the last two bins at <0.01. For event-time bins, Qwen's values are 0.33, 0.35, 0.46, 0.14, 0.20, which are not monotonically decreasing.
• FPS: Table 3 compares Object, Accident, Crime at 0.5, 1, 2 FPS. The authors state that higher FPS helps object perception but also increases context complexity, making risk prediction harder, so 1 FPS is kept by default. JoyVL's Accident PRS-U is 19.66 at 1 FPS and 6.54 at 2 FPS; MMDuet2's Accident is 0.00 at all three FPS.
• History frames: Table 4 sets the historical visual window to 1, 2, 4 frames or all. The authors state that retaining more historical visual information improves performance, and that current models are better at handling static historical information, with limited modeling of richer dynamic information. JoyVL's Accident PRS-U goes from 1.32 with 1 frame to 19.66 with all frames; MMDuet2's Accident remains 0.00.

Post-training

• What was measured: Table 5 compares Qwen3-VL-4B-Instruct, plus SFT, plus RL (P3I-Assistant-4B), and JoyVL. SFT uses 12,000 general samples plus 1,368 task samples; RL uses 863 rollout windows. The authors state that task samples are about 1.3K, total training samples about 13K, roughly 300 times fewer than JoyVL's 4M.
• Results: Average PRS-B goes from 28.62 to 36.72 after SFT, then to 42.12 after RL. Accident PRS-B goes from 41.78 to 61.78, then to 87.59. Intervention PRS-B is 9.13, 8.05, 9.02. After SFT, several PRS-U decrease: Event from 14.84 to 5.84, Accident from 30.93 to 2.96, Crime from 35.29 to 16.84.
• Author interpretation: The authors state that both types of training bring average PRS-B improvements, corresponding to +8.1 and +13.5, respectively; and that despite no corresponding training data, the gains generalize to Object Perception and Crime Alert. In Table 5, the final model's Crime PRS-B is 51.21, lower than JoyVL's 52.19. The authors state that complex responses like Intervention still require finer data construction and training strategies.

Other Ablations and Analyses

• Table 8: After removing the redundancy division, GPT-5.6-Luna's average PRS-B′ is 54.39, Qwen3-VL-4B-Instruct's is 52.18; VideoChat3's Intervention and Overall are N/A.
• Figure 5: JoyVL's zero-score proportions across duration bins are 50%, 97%, 71%, 85%, 88%; videos without GT responses are excluded from duration analysis.
• Appendix D: Qwen3-VL-8B-Instruct outputs 13 times between 4s and 277s for a single request; GPT-5.6-Luna's Warning is at 40.200s, GT is 18.867s, with the authors noting a difference of 21.333s; JoyAI-VL-Interaction only repeats the floor-arrival notification during the 35–39s correction window.
• Table 3: Qwen3-VL-4B-Instruct's Object PRS-B is 42.55 at 2 FPS, higher than 33.82 at 1 FPS; its Accident PRS-B drops from 41.78 to 30.02.
• Table 4: Qwen3-VL-4B-Instruct's Accident PRS-U at 2 frames is 11.54, lower than 12.58 at 1 frame and also lower than 30.93 with all frames.
• Table 2: Qwen3.5-9B's Accident PRS-U and PRS-B are both 15.82, lower than Qwen3.5-4B's 27.92; the paper does not explain this exception.

Limitations and Future Directions Noted by the Authors

• The paper does not have dedicated Limitations, Discussion, or Future Work sections. The Conclusion only expresses hope that future research will support proactive video assistants in dynamic, real-world scenarios, without listing specific shortcomings in data, models, or evaluation.
• In the Reproducibility Statement, the authors plan to release the benchmark and training data, construction pipeline, model checkpoints, training and evaluation code, prompts, and configurations after publication. This is a release plan, not a statement of limitations on current results.

Experimental Coverage

• The main evaluation includes 8 inference-time-adapted offline MLLMs and 4 native proactive streaming models, with metrics PRS-U and PRS-B (Section 4.1–4.2, Table 2).
• The evaluation data is P3IBench: 533 queries, 429 videos, 836 GT response targets, with tasks Event Monitoring, Object Monitoring, Accident Warning, Crime Alert, Event Intervention (Section 3.2, Figure 2).
• The models for FPS and history frame experiments are JoyVL, MMDuet2, Qwen3-VL-4B-Instruct, reporting tasks Object, Accident, Crime (Table 3, Table 4).
• Post-training starts only from Qwen3-VL-4B-Instruct; SFT includes 12,000 general samples and 1,368 task samples, RL includes 863 rollout windows, results in Table 5 and Appendix C.
• The paper does not report the number of evaluation repetitions, the model name for LLM review of the intervention subset, or the consistency values of the boundary estimates of the two vision-language models in Event Monitoring (Section 3.1, Section 3.3).

P3IBench is used to evaluate proactive video assistants in real streaming scenarios: not only understanding the current frame, but also deciding whether to respond and when to respond.

• Problem: The authors believe existing benchmarks focus on perception, while early warning and goal-conditioned intervention also require precise time boundaries. Responding too late is useless, and triggering without sufficient evidence is untenable.
• Benchmark: 533 queries from 429 videos, divided into daily event and object monitoring, traffic accident and crime-related warnings, and multiple interventions in five self-recorded scenarios. Mixed annotation combines multiple LLMs and human verification. PRS-U only matches responses to already-appeared targets, while PRS-B allows matching to the temporally nearest target; both score based on content correctness and time deviation, and penalize repeated responses.
• Main results: Among 12 models, GPT-5.6-Luna has an average PRS-B of 46.59 and average PRS-U of 38.78 (Table 2). Its Accident Warning and Intervention PRS-B are 74.64 and 29.85. MMDuet2 has an average PRS-B of 8.86, with both Accident scores at 0.00. The authors state that offline adapted models tend to repeat responses, while native proactive models tend to miss responses or delay.
• Training: After SFT and GRPO on Qwen3-VL-4B-Instruct, P3I-Assistant-4B achieves an average PRS-B of 42.12 and Accident PRS-B of 87.59 (Table 5). Intervention PRS-B is 9.02, not improved from 9.13 before fine-tuning. The authors state that average scores improve, but gains are uneven across capabilities.
• Author conclusion: Proactive assistance requires calibrated decisions on whether to respond and when, in addition to visual understanding. The authors hope these evaluation and training results support future research on proactive video assistants in dynamic real-world scenarios.