IH-Challenge: A Training Dataset for Improving the Instruction Hierarchy of Frontier Large Language Models

Core question: how to train LLMs to follow the instruction hierarchy (IH) in a generalizable and adversarially robust way.

• Problem setting: IH (Wallace et al., 2024) specifies that conflicting instructions are resolved by priority system ≻ developer ≻ user ≻ tool. The authors argue that robustly following IH can defend against jailbreaks, system prompt extraction, and direct and indirect prompt injection.
• Existing difficulties: The authors point out that IH failures are easily confused with instruction-following failures; conflicts may be subtle; and models may learn shortcuts, such as overrefusing all tasks involving passwords.
• Core hypothesis: If the training tasks themselves are simple (IF-simple), can be objectively judged by a Python scorer, and the task families are sufficiently diverse, then IH robustness trained by RL can generalize to attacks and task domains outside the training distribution.
• Paper contributions: Propose IH-Challenge—an RL training dataset—and fine-tune GPT-5-Mini with online adversarial example generation; the authors report that IH robustness on 16 benchmarks rises on average from 84.1% to 94.1%, and they open-source the dataset and scoring code.
• Threat model (the attacker setting in training and evaluation):
  • Goal: generate a low-priority conflict message that causes the defender to violate high-priority instructions and receive a low reward.
  • Knowledge: the attacker receives the high-priority instructions of the task skeleton as a reference; during training, the attacker is a frozen model without safety guardrails.
  • Capability: within a fixed tool-call budget, iteratively propose candidate messages, call the evaluation tool to query the current defender checkpoint and obtain scoring feedback, and then revise the attack.
  • Victim system: the defender model fine-tuned by RL (GPT-5-Mini and its fine-tuned version); evaluation also includes variants with system mitigations such as an output monitor.

Instruction hierarchy and prompt injection defenses

• Wallace et al. (2024)—proposed the concept of instruction hierarchy and trained models to resolve conflicts by role priority; this paper follows their framework and adopts the updated policy after OpenAI added the developer role.
• Wu et al. (2024)—proposed learning independent role embeddings added to token embeddings to achieve role separation.
• Hines et al. (2024)—proposed the sandwich defense, repeating high-priority instructions before generation; the authors state they found it also helps IH robustness.

Training-based IH defenses (directly compared by the authors)

• Zheng et al. (2025)—similar to this paper's method: treats the original instructions of IFEval (Zhou et al., 2023) as high-priority instructions, generates low-priority attack instructions that violate them, then does RL with a programmatically verifiable reward. The authors point out three differences: that dataset contains more IF-hard tasks rather than focusing on IH conflicts; it more easily leads to overrefusal; the attack strings are fixed and less adversarial. The authors also state that this paper verifies compatibility on large-scale frontier models and conducts adaptive evaluations such as human red-teaming.

RL training and reward hacking

• RL for controlling model behavior (Christiano et al., 2017), including online PPO (Schulman et al., 2017) and offline DPO (Rafailov et al., 2023).
• Reward hacking (Amodei et al., 2016; Gao et al., 2023): models find shortcuts to maximize reward; based on this, the authors choose to design rewards with a deterministic Python grader to reduce label noise.
• Nasr et al. (2025) and Wen et al. (2025)—fine-tune LLMs with RL for automated red-teaming; this paper reuses this strategy in §5.3 to conduct adaptive stress testing on GPT-5-Mini-R.

Balancing safety and helpfulness

• Askell et al. (2021)—points out that instruction-following training may push models to comply with malicious requests, while safety training makes models overly conservative toward benign requests; this paper evaluates this trade-off with dual safety/helpfulness metrics.

Baselines and benchmarks: the comparison for the main results is GPT-5-Mini before fine-tuning; capability comparisons include OpenAI o1 (chat win-rate). Evaluation benchmarks include TensorTrust, Gandalf Password, RealGuardrails, System IFEval, CyberSecEval 2, GPQA Diamond, AIME 2024, and multiple internal benchmarks. The authors position this paper as: on IH training data, simultaneously satisfying IF-simple, programmatic gradability, and resistance to shortcut learning, and verifying scalability to frontier models.

Overall approach: split IH training data construction into two stages, offline and online—offline ensures tasks are objectively gradable, online uses an attacker model to create adversarial conflicts, then RL trains the defender. IH-Challenge is the dataset produced by this pipeline.

IH policy and formalization

• Priority is system ≻ developer ≻ user ≻ tool (adopting the version after OpenAI added the developer role).
• Formalization: starting from the highest-priority role, maintain the feasible response set, B_i = B_i−1 ∩ C_i (if the intersection is non-empty), otherwise B_i = B_i−1; that is, low-priority instructions are followed only when compatible with high-priority constraints, and are ignored on conflict.

Offline task skeleton construction (§3.1)

• Skeleton contents: one or more high-priority instructions, placeholders for low-priority conflict messages, and a Python grader; the grader remains valid under any low-priority message.
• Three design principles: IF-simple (difficulty comes from the IH conflict rather than the task itself, and remains simple under adversarial inputs); programmatically gradeable (conflicts may only appear in low-priority roles, excluding tasks such as jailbreaks that cannot be objectively graded); avoid shortcut learning (diverse task structures to prevent "refuse whenever a password is seen"-style shortcuts).
• Construction pipeline: LLM synthesizes structured task objects (dialogue, grader code, sample responses that should pass/fail) → static checks (e.g., AST parsing) and format normalization → unit-test the grader with the samples → human review. The authors state that human review still does not guarantee the grader is fully correct.
• Four task families (Table 1): Single-Constraint (19 IFEval-style single-constraint tasks), Multi-Constraint (combinations of 2–6 atomic constraints), Input-Conditioned (parse messages according to a specified schema and output in a strict format), Anti-Overrefusal (rewrite benign requests to look violating; refusal counts as failure).

Online adversarial conflict synthesis (§3.2)

• Attacker: frozen model without safety guardrails, able to call evaluation tools; the goal is to produce, within a fixed tool-call budget, low-priority messages that make the current defender receive a low reward.
• Budgeted propose–evaluate–revise loop: propose candidate messages → tools fill the messages into the skeleton, query the defender, score with the Python grader and return pass/fail and diagnostics → the attacker revises based on feedback; after the budget is exhausted or early stopping, output the final message.
• Attack diversity: structured attack prompts provide IH background and require exploring multiple attack strategies; although the attacker is frozen, because feedback targets an ever-stronger defender, attacks become harder over training.
• Static evaluation set: run the same synthesis pipeline on held-out skeletons against held-out defender checkpoints to obtain fixed conflict prompts for measuring generalization.

Defender training

• Each training step first synthesizes conflict prompts online using the above attack rollouts, samples multiple defender responses, scores them with the Python grader, and performs a policy gradient update.
• To avoid capability regression, IH data is mixed with a small number of capability tasks (e.g., tool use, writing). The paper does not report specific hyperparameters such as the RL algorithm name, learning rate, or tool-call budget.

实验设置

• 被测模型：GPT-5-Mini 与其在 IH-Challenge 上 RL 微调得到的 GPT-5-Mini-R；部分实验加入 output monitor 等系统缓解变体。
• 训练攻击者：在线合成用 frozen attacker LLM（无安全护栏）；§5.3 自适应自动红队用 anti-refusal GPT-5（经微调去除安全护栏）做攻击方，defender 冻结。
• 基准：IH-Challenge 四个 split；OOD 内部基准（Tutor Jailbreak、多轮冲突对话）；学术基准（Gandalf Password、TensorTrust、RealGuardrails、System IFEval）；OpenAI Production Benchmarks（安全/有用性）；Internal PI Benchmark 与 CyberSecEval 2（提示注入）；GPQA Diamond、AIME 2024、chat win-rate vs. o1（能力）。
• 指标与判定：robustness 为低优先级指令冲突时响应满足最高优先级约束的比例；IH-Challenge 用 Python 评分器；OOD 评测多用 LLM grader（具体模型论文未说明）；安全/有用性按 Yuan et al. (2025) 的协议打分。
• 人工红队：127 名美国远程红队成员，每人攻击 10 个"系统 × 任务类型"组合（19 种任务类型），成功攻击有奖金激励（多数组合赏金 $510）。
• 重复次数与样本量：多数基准未报告；人工红队各模型任务数为 265–271（Table 7）。

主结果

据 Table 3、Table 4、Table 5（数值为 robustness，括号内为相对 GPT-5-Mini 的变化）：

评测	GPT-5-Mini	GPT-5-Mini-R
System <> User Conflict（Table 3）	0.84	0.95 (+0.11)
Developer <> User Conflict（Table 3）	0.83	0.95 (+0.12)
TensorTrust (dev-user)（Table 4）	0.76	0.91 (+0.15)
TensorTrust (sys-user)（Table 4）	0.86	0.94 (+0.08)
RealGuardrails (Handwritten)（Table 4）	0.82	0.89 (+0.07)
System IFEval（Table 4）	0.92	0.96 (+0.04)
Internal PI Benchmark（Table 5）	0.44	1.00 (+0.56)
CyberSecEval 2（Table 5）	0.88	0.91 (+0.03)

• 作者称 RL 微调后在所有 OOD 评测上都有鲁棒性提升，且多数数据集的任务训练中未见、只能由 LLM grader 判定，因此提升"几乎完全来自泛化"。
• 摘要与引言报告 16 个基准平均从 84.1% 升至 94.1%；Internal PI Benchmark 被饱和（1.00）。
• Figure 4b 显示 held-out 攻击下 GPT-5-Mini-R 鲁棒性高于 GPT-5-Mini：Impersonation 0.23→0.90，Automated Attacks 0.73→0.94，Human Red-teaming 0.90→0.97（图上标注值）。

安全与有用性（Production Benchmarks）

• 测了什么：在 system prompt 加入各类别安全规范（safety spec）后，测各违规类别的 safety score 与 helpfulness score（Figure 5、6）。
• 结果：作者报告 GPT-5-Mini-R + safety spec 在所有类别上 safety score 高于 GPT-5-Mini（无论是否加 spec）；有用性基本保持。摘要报告不安全行为从 6.6% 降到 0.7%。
• 作者解读：安全违规请求来自低优先级指令、与安全规范冲突，IH 鲁棒性提升使模型更善于解决这类冲突；附录 A.3 称用全局安全规范也能得到类似提升。

自适应红队（自动与人工）

• 自动红队（§5.3）：用 IH-Challenge 反向奖励 RL 训练攻击者对抗冻结的 GPT-5-Mini-R；Figure 8 显示三个 split 上 ASR 随训练从低位逐渐上升，加 output monitor 后各 split 的 ASR 都更低（图上终点标注：Single-Constraint 约 0.15 vs 0.20，Multi-Constraint 约 0.10 vs 0.14，Input-Conditioned 约 0.3 vs 0.5）。作者认为 defense-in-depth 对自适应攻击者有益。
• 人工红队（§5.4，Table 7）：攻击成功率 GPT-5-Mini 36.2% → GPT-5-Mini-R 11.7% → GPT-5-Mini-R + Monitor 7.1%；平均每任务尝试次数 32.84 → 52.39 → 43.08。作者称红队对更强的模型投入更多努力仍更少成功。

其他消融与分析

• 任务 split 消融（Table 6）：用全部四个 split 时 robustness 0.941、overrefusal 0.950、safety 0.957、helpfulness 0.773；去掉 Anti-Overrefusal 后 overrefusal 降到 0.831、helpfulness 降到 0.613；只用 Multi-Constraint + Input-Conditioned 时 robustness 0.924。
• 系统缓解（§5.2，Figure 7）：对 GPT-5-Mini，output rewriter、sandwich 等能提升鲁棒性；对 GPT-5-Mini-R，多数缓解失去效果，有些反而鲁棒性和能力都更差。
• 过度拒答与能力（Table 2）：IH-Challenge (overrefusal) 0.79→1.00；TensorTrust (overrefusal) 0.91→0.90；GPQA Diamond 0.83→0.83；AIME 2024 0.93→0.94；Chat WinRate vs. o1 0.71→0.66；User Preference Score 0.46→0.40。
• 例外：TensorTrust (overrefusal)、System <> Developer Conflict（0.86→0.86）、Gandalf Password (sys-user)（0.99→0.99）无提升；chat win-rate 与 preference score 有小幅回退。

Limitations and future directions noted by the authors

• Simultaneous attacker-defender training not explored (Conclusion): the authors propose that training the attacker and defender simultaneously is a promising compute-scaling direction, but this paper only does two settings: fixed attacker training the defender, and fixed defender training the attacker.
• Risk of reward drift (Conclusion): the authors argue that without programmatically gradable rewards, the attacker and defender objectives may deviate from the true robustness objective, and speculate that such rewards may be a necessary component for scaling this adversarial training paradigm.
• Grader correctness (§3.1): the authors state that filtering and unit tests can remove almost all cases where the task description and grader are inconsistent, but do not guarantee the grader is fully correct, so they performed human review.
• Limited task diversity (§6): the authors point out that IH-Challenge tasks are IF-simple and, because objective programmatic grading is required, have limited diversity; robustness generalization depends on the OOD generalization ability of RL.
• Appendix A.4: gives two examples of LLM judge misjudgments (the defender actually followed IH but was judged as a successful attack), illustrating sources of error in LLM judging.

Scope of experiments

• The models fine-tuned and evaluated are GPT-5-Mini and GPT-5-Mini-R, two in total; capability comparisons include o1 (chat win-rate).
• Evaluation coverage: four IH-Challenge splits, 3 types of held-out attacks, 4 academic IH benchmarks, internal OOD benchmarks, OpenAI Production Benchmarks safety/helpfulness, 2 prompt injection benchmarks, 4 capability and overrefusal evaluations, and both automated and human adaptive red-teaming.
• Human red-teaming scale is 127 red-team members, 265–271 tasks per model (Table 7).
• The paper does not report RL training hyperparameters (algorithm, steps, learning rate), the numerical tool-call budget of the attacker, or the sample sizes and number of repetitions for each benchmark.
• The paper does not state the specific models of the LLM graders used for OOD evaluation and safety evaluation, nor does it report the agreement between graders and human judgments.

OpenAI 提出 IH-Challenge，一个用于训练 LLM 鲁棒遵循指令层级（system ≻ developer ≻ user ≻ tool）的 RL 数据集，目标是让 IH 行为跨任务域泛化且抗对抗。

• 要解决的问题：IH 是防御越狱、系统提示词提取和提示注入的统一策略，但作者认为其难以训练——失败易与指令遵循失败混淆、冲突微妙、模型易学到过度拒答等捷径。
• 方法要点：离线构造含高优先级指令与 Python 评分器的任务骨架（Single-Constraint、Multi-Constraint、Input-Conditioned、Anti-Overrefusal 四族），保证 IF-simple、可客观判定、防捷径；训练时由无护栏的冻结攻击模型在预算内迭代探测 defender、在线合成对抗冲突消息；再用评分器奖励做 RL，并混入少量能力任务。
• 主要结果：作者报告 GPT-5-Mini-R 在 16 个基准上 IH 鲁棒性平均从 84.1% 升至 94.1%；自适应人工红队攻击成功率从 36.2% 降到 11.7%（加 output monitor 后 7.1%）；Internal PI Benchmark 从 0.44 升至 1.00；加安全规范后不安全行为从 6.6% 降到 0.7% 且有用性基本保持；过度拒答评测从 0.79 升至 1.00。
• 代价与例外：chat win-rate vs. o1 从 0.71 降到 0.66，preference score 从 0.46 降到 0.40；个别评测（如 System <> Developer Conflict）无提升。
• 作者结论：作者认为 IH 是同时获得多项安全收益的机制，RL 配合设计良好的代理任务可泛化到无法程序化判定的领域（如安全）；并提出攻防同时训练的对抗训练范式是后续方向，推测可程序化奖励可能是其必要组件。数据集与评分代码已开源。