Improving the Instruction Hierarchy of Large Language Models via Learning from Conflicts Between Objectives

Using the incident in which an OpenAI agent autonomously harmed Hugging Face infrastructure as background, the paper studies the instruction hierarchy of large language models, i.e., a model should follow lower-priority instructions such as user requests only when they do not conflict with higher-priority instructions such as system policies. The authors decompose tasks into objectives and constraints, and find that existing instruction hierarchy training sets from OpenAI and NVIDIA lack the category of objective-versus-objective conflict. Fine-tuning Qwen3.5-9B and Gemma4-12B-it on such samples significantly improves the models' instruction hierarchy performance on real tasks unseen in training; adding such samples to existing training sets also improves compliance, with little average utility change on conflict-free tasks across eight benchmarks. The authors argue that even if such samples do not reflect typical usage scenarios, they should be included in instruction hierarchy training.