StepSafe: Learning Timely Intervention with Context-Aware Safety Reasoning

Agent safety monitoring must intervene at the correct step before harm becomes irreversible, yet most existing objectives only supervise whether an action is unsafe.

• Scenario: The authors state that large language model agents call tools, move data, and collaborate with other agents; tool calls, data transfers, or delegations can cause harm, such as handing personal records to an external party. A monitor must decide before each proposed action whether to allow it.
• Existing shortcomings: Trajectory-level auditors judge completed trajectories. The introduction states that Safiron, TS-Guard, and StepGuard use adjudication and risk rewards for reinforcement learning, learning whether an action is unsafe rather than where risk first becomes visible, or the last step at which intervention can still prevent harm.
• Three challenges: The authors state that existing labels are mostly trajectory-level safe/unsafe, rarely saying which step should be intervened on or which earlier evidence supports it; trajectory-level rewards penalize correct CONTINUE before risk appears, while step-level rewards are dominated by many routine safe steps; safe steps have no annotatable content, so when not to intercept cannot be learned.
• This paper's approach: Propose Adaptive Stage-Aligned Policy Optimization (ASPO) to train the step-level reasoning guardrail StepSafe. Training data carries violation classifications and intervention windows, augmented with multi-agent red-teaming.
• Threat model:
  • Goal: Output CONTINUE, ASK, or STOP before execution, with a rationale. CONTINUE allows, ASK pauses for clarification, and STOP blocks the action.
  • Visible information: Receives only the task, access policy, bounded history, and current proposed action; timing must not rely on unseen future events.
  • Capability: In training replay, CONTINUE advances to the next step record, while ASK or STOP ends that monitoring episode. In evaluation, STOP, or ASK with risk at least 50, counts as intervention.
  • Victim system: An agent that executes multi-step tasks, calls tools, and may delegate. Unsafe pressure can come from user requests, external content, tools, peer agents, or stored memory, among others. The paper does not state whether the attacker is white-box, black-box, or no-box.

• Content Guardrails and Agent Risk: Inan et al. (2023)'s Llama Guard, Zeng et al. (2024)'s ShieldGemma, Han et al. (2024)'s WildGuard, and Zhao et al. (2025)'s Qwen3Guard mainly classify harmful content in prompts or replies. Andriushchenko et al. (2025), Zhang et al. (2025; 2024), Ruan et al. (2024), Zhan et al. (2024), and Debenedetti et al. (2024) discuss risks that depend on evolving execution context and can also arise from sequences of seemingly benign messages and actions.
• Trajectory-Level and Step-Level Monitoring: Luo et al. (2025a)'s AgentAuditor and Liu et al. (2026b)'s AgentDoG judge completed interactions. Proactive monitoring includes Huang et al. (2026)'s pre-execution plan evaluation, Chen et al. (2026)'s SafePred, Li et al. (2026a)'s trajectory-prefix risk tracking, and Luo et al. (2025b)'s adaptive safety checks. Mou et al. (2026)'s ToolSafe detects unsafe tool calls and gives feedback; Zheng et al. (2026)'s StepGuard uses prefix-aligned supervision and reinforcement learning to balance over-defensiveness and under-defensiveness. The introduction specifically points to Safiron, TS-Guard, and StepGuard as learning only whether something is unsafe, not when to act.
• Reinforcement Learning Beyond Outcome Rewards: Shao et al. (2024)'s GRPO assigns outcome-based shared advantages to all tokens in a rollout. Feng et al. (2026)'s GiGPO estimates step-level advantages based on a shared anchor state. Loye et al. (2026)'s RUBAS uses multidimensional safety rulebooks to reward complete agent trajectories, aligning the agent itself rather than deciding when the monitor should intervene; Gunjal et al. (2026) use rulebooks to provide structured feedback. Lu et al. (2026)'s SDAR and Gu et al. (2026)'s RCSD use on-policy self-distillation for token-level guidance, with privileged context fixed before training.
• Baseline Methods: API models are Claude Sonnet 5, GPT-5.6 Sol, GPT-5.6 Terra, and GPT-5.6 Luna; open backbones include Qwen3-4B, Qwen2.5-7B-Instruct, Qwen3.5-4B, Qwen3-4B-Instruct-2507, and Qwen3-8B. Monitor baselines are AgentDoG, ShieldAgent, AgentAuditor, Qwen3Guard-Gen-8B, Llama-Guard-3-8B, StepGuard, SafePred, and TS-Guard.
• Benchmarks/Datasets: Evaluation uses ASSEBench Safety and Security, R-Judge, AgentHarm, and ATBench, totaling 4,033 trajectories. Training trajectories come from AgentDoG and HarnessAudit, then are expanded with red teaming.

The authors claim StepSafe treats intervention timing as an explicit learning objective; SAPO gives phase-dependent preferences on the same prefix, and AFSD lets privileged guidance evolve with the monitor's own false positives.

StepSafe在每个提议动作执行前读取可见前缀，输出简短推理和结构化裁决；ASPO用决策信用加推理token信用训练它，L_ASPO(θ)=L_SAPO(θ)+λ L_AFSD(θ)。前一项按整条决策给信用，后一项按token给信用。

问题设定

• 输入与输出：可见前缀含任务、访问策略、有界历史和当前提议动作。输出为推理轨迹加十字段裁决，主动作yt属于CONTINUE、ASK、STOP。
• 可审计字段：failure、fault、harm、source分别表示被违反的约束、过程错误、伤害和压力来源；另有对象、证据、澄清问题和timing。timing只相对可见前缀中的边界。
• 回放：沿记录轨迹逐步查询。CONTINUE进入下一步记录动作，ASK或STOP结束该次监控。每条轨迹采样G次；附录B写明G=8。

标注与红队

• 四轴分类：沿AgentDoG的诊断轴标注failure、fault、harm、source，类别数分别为12、10、10、8。定义见Table 7与Appendix A.1。标注模型为GPT-5.6 Luna。
• 干预窗口：不安全轨迹记录信号s、边界b和最后可阻止步h，且s≤b≤h。干预目标为s到h之间的步骤；安全轨迹全程为继续。地标同时指定阶段。
• 红队：每轮用训练失误和覆盖缺口指导新样本。operations、security、social三类人格从种子池写任务，轨迹生成器写出轨迹及其良性孪生，审稿角色核验后重放。式(4)只保留审稿确认的样本。三轮共512对，其中328对来自失误、184对来自覆盖缺口；孪生在边界前记录相同，只改权限、接收方、来源事实或动作之一。红队角色由GPT-5.6担任。红队样本只进训练集。附录A.2、A.4给出标注与流程说明。

SAPO

• 阶段分数：风险可见前奖励CONTINUE、惩罚ASK和STOP。澄清阶段s≤t<b时ASK为+1、STOP为+0.5。边界、边界之后到h之前，以及最后可阻止步，奖励STOP；最后一步对ASK为−1。畸形裁决得−1，h之后的步骤不进入训练。附录B.2另写格式或字段违规的惩罚为0.1。
• 同前缀比较：只与同一可见前缀上的其他回复比较，A_i=σ_i−1/(|G(x)|−1)∑_{k∈ G(x)∖{i}}σ_k。仅一条回复到达该前缀时优势为0；分数相同的组不更新。作者不做法内标准差归一化，以保留ASK与STOP之间的较小偏好。
• 类预算：按标注把步骤分成应继续和应干预两类，两类都出现时各得一半平均权重，不按回复条数平分。更新使用GRPO的裁剪目标，另加KL项。

AFSD与训练

• 特权教师：教师与学生同参数。干预步的教师看到标注裁决和该步违规描述；安全步看到CONTINUE、可用的继续理由，以及从学生误报挖出的规则。梯度只过学生。
• 误报规则：同一键至少有来自两条轨迹的三个案例才候选规则。规则须清掉至少一半来源误报，且不能清掉同键的已知违规，包括良性孪生的不安全对应样本。未通过则带反例重写一次再测。量规最多64条，按上置信界检索，探索常数0.5。
• 训练设置：骨干为开启thinking的Qwen3-4B或Qwen3-8B。先在2,402个单步样本上监督2个epoch，再做500步强化学习，按验证集J取最佳步。附录B写λ=0.01、门控β=5、学习率10−6、KL系数0.01、裁剪ϵ=0.2。评测贪心解码，回复最多2,048 token。

Experimental Setup

• Data: Five benchmarks totaling 4,033 trajectories. ASSEBench Safety, ASSEBench Security, R-Judge, and AgentHarm use AgentAuditor's release and protocol; ATBench uses AgentDoG's protocol. Each unsafe trajectory is additionally annotated with the harm step.
• Models Under Test and Baselines: The paper's models are StepSafe-4B and StepSafe-8B. Baselines include Claude Sonnet 5, GPT-5.6 Sol/Terra/Luna, and the open backbones and eight safety monitors in Table 1. AgentAuditor runs on Qwen3-4B. Published monitors use their original prompts, label mappings, and decoding; APIs and untrained backbones use the paper's monitoring prompt.
• Metrics: A trajectory is flagged if any step intervenes. Intervention is STOP, or ASK with risk≥50. Acc and mF1 are relative to trajectory-level ground truth. ASR is the proportion of unsafe trajectories with no intervention at or before harm step h; Utility is the proportion of safe trajectories never flagged. J=recall+Utility−1, all metrics ×100. Timely means intervention at h or one step before; two or more steps early is premature. BLE is the average step distance from first STOP to boundary b.
• Sample Sizes: Timing analysis uses 1,260 unsafe trajectories from ASSEBench Safety, Security, and R-Judge that have at least one agent action before the harm step. Paired evaluation is 327 pairs across 36 tasks. Appendix D.1 uses three seeds; the main reinforcement learning seed is 42. The paper does not report the number of evaluation repetitions per trajectory; StepSafe evaluation is one greedy sample per step.

Main Results

According to Table 2. The table shows timely intervention and utility aggregated over five benchmarks; J matches the total column in Table 1.

Swipe horizontally to read the table

Guardrail	ASR↓	Utility↑	J↑
StepSafe-4B	27.6	80.0	53.0
StepSafe-8B	21.7	74.8	54.4
StepGuard	32.7	78.5	46.2
AgentDoG	8.2	41.3	34.8
ShieldAgent	4.7	34.7	30.8
Qwen3Guard-Gen-8B	75.6	48.7	−7.6
GPT-5.6 Sol	18.0	82.5	66.6
Claude Sonnet 5	22.2	88.5	64.0

Omitted Table 2 rows include GPT-5.6 Terra (ASR 19.0, Utility 76.6, J 57.6), GPT-5.6 Luna (21.0, 74.9, 55.7), TS-Guard (39.4, 75.8, 37.2), SafePred (36.1, 65.2, 30.1), Llama-Guard-3-8B (47.0, 80.9, 28.9), and Qwen3-4B thinking on (33.3, 64.1, 28.5).

• Authors' Overall Assessment: The authors claim StepSafe achieves SOTA among open-source monitors, with total J of 53.0 and 54.4 for 4B and 8B, compared to StepGuard's 46.2; and claim it ranks first on three of the five benchmarks. Table 1 counterexamples: on R-Judge, StepGuard-4B has J 69.3, StepSafe-4B 68.0, and 8B 54.6; on AgentHarm, ShieldAgent-7B has J 97.3, StepSafe-4B 64.8.
• Scale and Frontier Models: From 4B to 8B, ASR goes from 27.6 to 21.7, Utility from 80.0 to 74.8. The authors claim 8B's ASR falls within the 18.0–22.2 range of four frontier models and is lower than Claude Sonnet 5's 22.2; API models still have J 55.7–66.6. The authors claim 4B's Utility is higher than GPT-5.6 Terra and Luna, lagging only in preventing violations, and is dominated only by Claude Sonnet 5 and GPT-5.6 Sol.
• Per-Benchmark: The authors claim gains are largest on ASSEBench Safety and Security, with R-Judge favoring 4B more. In Table 1, StepSafe-4B/8B J values are: Safety 44.0/49.3, Security 66.9/72.3, R-Judge 68.0/54.6, AgentHarm 64.8/64.2, ATBench 43.2/43.5.

Intervention Timing

• What Was Measured: On 1,260 unsafe trajectories, the position of the first intervention relative to the last preventable step h was compared. In Figure 4, StepSafe is StepSafe-8B.
• Results: The authors claim the proportion intervening exactly at the harm step is 29.3%, with the open-source baseline SafePred at 22.1%; missed detections are 10.9%, which the authors claim is lower than other monitors except AgentDoG and ShieldAgent. The authors claim that among other monitors, the largest share of premature interventions comes from one action before the harm step.
• Authors' Interpretation: The authors claim the phase schedule lets the model continue when risk is not yet visible and stop when harm is imminent, rather than intercepting early. In Table 5, the false-positive rulebook relative to SAPO alone changes the proportion at the harm step from 26.9% to 29.7%, two-or-more steps early from 22.8% to 19.5%, Utility from 79.1 to 80.0, and BLE from 2.10 to 1.94. The authors claim StepSafe stops 59.1% of unsafe trajectories within one step before an irreversible action executes.

Training Signal Ablation

• Stages: In Table 3, Qwen3-4B has J 28.5; adding SFT gives J 46.2, ASR 35.4, Utility 81.8; adding SAPO gives J 51.9, ASR 27.2, Utility 79.1; adding AFSD (i.e., StepSafe) gives J 53.0, ASR 27.6, Utility 80.0.
• Same-Task Pairs: On 327 pairs, correct discrimination goes from 70.9 with SFT to 68.2 with SAPO, then to 75.5 with AFSD. The authors report this paired accuracy is higher than SFT and SAPO, p<0.01. SAPO catches more unsafe trajectories missed by SFT but also flags more of their safe twins; AFSD removes these false positives while retaining caught violations.
• Credit Placement: Table 4 uses Qwen3-4B without SFT. Trajectory-level GRPO has ASR 31.5 and J 39.6; step-level GRPO has 34.8 and 34.8; GiGPO has 38.3 and 38.8; SAPO has 26.4 and 41.2. The authors claim denser rewards alone are insufficient; same-prefix comparison is what keeps credit on the current decision.

Other Ablations and Analyses

• Teacher Context (Table 5): Adding only violation feedback drops Utility from 79.1 to 77.1; adding the false-positive rulebook brings Utility to 80.0 and BLE to 1.94.
• Red Teaming (Table 6): Removing red team gives Acc 75.8, ASR 28.3, Utility 78.4, J 51.7; full StepSafe-4B has 76.5, 27.6, 80.0, 53.0.
• Binary Reward vs. Phase Schedule (Table 15, no SFT): J is 41.0 vs. 41.2, ASR 27.8 vs. 26.4; proportion stopping exactly at the boundary (BIR) is 26.5 vs. 30.7, and the share of trajectories with STOP is 62.9 vs. 71.7.
• Full-Step SFT (Table 13): Full-step step-level SFT has ASR 67.1 and J 24.5; class-balanced gives ASR 38.7 and J 40.1; multi-round SFT has Utility 70.7 and J 44.6. All three have J lower than the key-step warm start's 46.2.
• Scale, Seeds, and Operating Points: In Table 11, untrained Qwen3-8B has J 35.9 and ASR 22.7. Table 10 shows StepSafe-4B J 52.9±0.8 across three seeds. Table 12, changing the correct CONTINUE reward from 1 to 1.25, gives ASR 32.4, Utility 84.3, and J 48.3.
• Coverage and Exceptions: Appendix A.5 states that after red teaming, reporting fault share increases 7.9×, state-tracking 3.6×, and incident-response failure 3.2×; environmental harm and flawed governing specifications remain largely uncovered. In Appendix A.3, across 14 monitors, the correlation between J and Utility−ASR is r=0.982.

Limitations and Future Directions Noted by the Authors

• Offline Utility: Utility measures the absence of scored alerts, not whether the online task is completed (Limitations).
• Label Source: Step-level labels are generated by language models and verified. The authors state that annotation was done in two passes—one for annotation, one for confirmation and revision—to provide training and evaluation standards (Limitations, pointing to Appendix A.3).
• Reporting Scope: Because annotation is generative, the authors state they only treat accuracy and J based on ground-truth labels as objective evaluations, leaving step-label-dependent evaluations in ablations, and analyze the correlation between step-level labels and ground-truth labels (Limitations).
• Future Benchmarks: The authors consider it worthwhile to build benchmarks with step-level annotations, longer trajectories, and multi-agent collaboration in real agent scenarios (Appendix C).

Experimental Coverage

• Evaluation Trajectories: Five benchmarks totaling 4,033 trajectories: ASSEBench Safety, ASSEBench Security, R-Judge, AgentHarm, and ATBench (§3.1).
• Timing and ASR: Comparison of first intervention relative to the harm step uses 1,260 unsafe trajectories from the first three benchmarks (§3.2). The paper states ASR is annotated on ASSEBench Safety, Security, R-Judge, and AgentHarm (Appendix C).
• Baselines and Training: Table 1 compares four API models, open backbones, and eight safety monitors. Training trajectories come from AgentDoG and HarnessAudit, plus 512 red team pairs; red team trajectories are used only for training, and the evaluation sets do not include them (Appendices A.4, B).
• Ablations: The main text compares training stages, credit estimators, AFSD teacher context, and red team data; Appendix D reports three seeds, 4B and 8B, operating points, and a comparison using all annotated steps for SFT.
• Annotation and Unreported Items: Landmarks and verdicts are proposed by GPT-5.6 Luna, with an unnamed verification model; the correlation between J and Utility−ASR is r=0.982 (Appendix A.3). The paper does not report agreement between human annotators and labels, nor online task completion rates.

StepSafe is a pre-execution, step-level reasoning guardrail that uses ASPO to learn when to continue, ask, or stop.

• Problem: Agent harm depends on actions already taken, tool outputs, and interaction. The authors argue that existing step-level monitors can judge whether an action is unsafe, but cannot learn from outcome labels when risk becomes visible, up to which step intervention can still prevent harm, or when a seemingly dangerous step can be allowed.
• Method: Before each proposed action, the model sees only the task, policy, bounded history, and current action, and outputs CONTINUE, ASK, or STOP with classification fields. SAPO scores by signal, boundary, and last preventable step, compares only with other replies sharing the same prefix, and gives fixed update shares to continue and intervene classes. AFSD lets the model act as teacher: intervention steps use annotations, and safe steps use rules filtered from its own false positives. Data is annotated on AgentDoG and HarnessAudit trajectories and augmented with 512 red-team sample pairs.
• Results: Across five benchmarks totaling 4,033 trajectories, StepSafe-4B has total J of 53.0, ASR of 27.6, and Utility of 80.0; 8B has 54.4, 21.7, and 74.8 (Table 1, Table 2). The authors state this is SOTA among open-source guardrails, with StepGuard's total J at 46.2. In Table 1, R-Judge favors 4B more (J 68.0 vs 54.6), and on AgentHarm, ShieldAgent-7B has J of 97.3. On 1,260 unsafe trajectories, the authors state 8B intervenes exactly at the harm step in 29.3% of cases, with 10.9% missed detections.
• Author conclusions: The authors state the staged schedule teaches the model to continue when risk is not visible and to stop when harm is imminent; the false-positive rules teach it when not to intervene. The authors also note that offline Utility is not online task completion, and step-level labels are generated and verified by language models.