Instruction Hierarchy: Training Large Language Models to Prioritize Privileged Instructions
The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions
Translation in progress — untranslated text is omitted.
OpenAI researchers propose an instruction hierarchy training method that makes large language models, when instructions conflict, prioritize following system prompts first, then user messages, and finally third-party content, in order to defend against prompt injection, jailbreak, and system prompt extraction attacks. The method combines synthetic data generation with context distillation: aligned lower-privilege instructions train the model to follow them as usual, misaligned instructions train the model to ignore or refuse them, and an automated red-team model is used to generate injection data, with supervised fine-tuning and RLHF on GPT-3.5 Turbo. Evaluations show improved robustness across all attack scenarios, with a 63% improvement in system prompt extraction defense; the training data deliberately contains no jailbreak samples, yet the model's robustness to unseen attacks such as jailbreaks still improves by over 30%, and general capabilities are largely unaffected, but there is regression on two over-refusal evaluations.
- LLMs often treat system prompts and untrusted user or third-party input as having equal priority, allowing attacks such as prompt injection, jailbreaks, and system prompt extraction to override the original instructions, threatening LLM-based agent applications.
- The authors propose an instruction hierarchy: system messages take precedence over user messages, and user messages take precedence over third-party content. They use context synthesis to handle aligned instructions and context ignorance to handle conflicting instructions to generate training data, and perform SFT and RLHF on GPT-3.5 Turbo.
- The authors report that after training the model's robustness improves on all main evaluations, with system prompt extraction defense improving by 63%; there is also generalization to attacks deliberately held out during training (jailbreak, tool injection, password extraction), with improvements up to 34%; over-refusal is close to baseline on most evaluations, with declines on two adversarial evaluations.
- The authors note that the current model uniformly ignores instructions in browsing and tools, handles only the text modality, and may still be vulnerable to strong adversarial attacks, and list these as future directions; the experiments were conducted only on GPT-3.5 Turbo, and the paper does not report the sample size or number of repetitions for each evaluation.
The paper proposes the instruction hierarchy training method and provides details of the data construction, validating on GPT-3.5 Turbo improved robustness against attacks such as prompt injection.
Related events and news
0 events · 4 news mentions
News that mentions this paper
- evantocites the paper's id or link谁会服从注入的指令？21 个语言模型的提示注入测试
Wallace et al. (OpenAI, 2024): The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions
- Anthropic Alignment Sciencecites the paper's id or linkAnthropic 研究：自动化对齐研究者在小模型上缓解十类对齐失败
(4) Wallace et al. (2024), Instruction Hierarchy (arXiv:2404.13208): our selective-compliance guard is an honesty analogue that routes on prefix content. (5) Geva et al. (2021,
- OpenAI Alignment Researchcites the paper's id or linkOpenAI 披露可自我复制的提示注入，基于 GPT-5.4-mini 与 GPT-5.5 内部检查点
Wallace, E., Xiao, K., Leike, R., Weng, L., Heidecke, J., & Beutel, A. (2024).The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions. arXiv preprint.
Show 1 more news items
- Tech Policy Presscites the paper's id or linkAI 安全不能只靠约束系统提示词，还需评估模型实际输出
A model may be instructed to act as a helpful assistant, or to “not provide information for creating harmful substances or weapons.” These natural language instructions are often more accessible and