Can dilemmas and conflicts in LLM alignment be resolved? A perspective from priority graphs

LLM alignment requires choosing among instruction and value priorities that change with context and may be inconsistent.

• Scenario: The authors state that as models move from content generation to autonomous agents, they simultaneously encounter system instructions, user instructions, retrieved information, and multiple human values. Section 2 states these conflicts are widespread in real use, not edge cases.
• Existing work: The authors state that prior work has begun to identify everyday value dilemmas (Chiu et al., 2025a) and hierarchical instruction conflicts (Wallace et al., 2025; Zhang et al., 2025; Wu et al.), and that the latter degrades current model performance. The authors extend the taxonomy to information conflicts, classical ethical dilemmas, and subjective preferences.
• Core observation: Choices can be written as conditional distributions, and priorities can be drawn as a directed graph. The authors state this graph is neither static nor necessarily consistent across contexts, and may contain directed cycles, making unified, stable alignment difficult.
• What this paper provides: a taxonomy of five conflict types, a priority graph, a manipulation method called priority hacking, and runtime verification of context against external sources. The authors also argue that many ethical and value dilemmas are philosophically irreducible.
• priority hacking:
  • Goal: make an aligned model bypass the safety constraint Asafety to complete a harmful request packaged as a higher-priority value.
  • Knowledge: the paper does not use white-box, black-box, or no-box. The authors state the adversary exploits value hierarchies already formed during training, seeking a benign high-level value Avalue that already takes precedence over safety constraints in some context.
  • Capability: construct a context Cadv such that Avalue takes precedence over Asafety in that context. Section 4 places the harmful request in a narrative of upholding justice; the prompt is not reproduced here.
  • Victim system: an aligned LLM that should refuse dangerous queries under normal conditions; Section 5 describes the verification target as an LLM agent that executes instructions based on user context.

The paper does not have a dedicated Related Work section. The introduction and Section 2 cite related work by conflict type, and each group below does not exceed the scope actually covered by the paper.

Instruction hierarchy and competing goals

• Wallace et al. (2025) discuss training LLMs to prioritize privileged instructions; Zhang et al. (2025)'s IHEval evaluates whether models follow instruction hierarchies; Wu et al. (citation without year) use instructional segment embedding to apply instruction hierarchies to safety. The authors state that conflicts between developer instructions and end-user instructions cause performance degradation in current models.
• Wei et al. (2023), Perez and Ribeiro (2022), Schulhoff et al. (2023), Shen et al. (2023), Qiang et al. (2023), and Yong et al. (2023) are grouped into the same phenomenon: safety instructions can compete with complex format constraints given by users. The paper only describes this phenomenon and does not provide quantitative comparisons with these methods.

Information conflicts and external inputs

• Xie et al. (2023) analyze LLM behavior in knowledge conflicts; Xu et al. (2024) survey knowledge conflicts in LLMs. The authors use these to illustrate that parametric knowledge and external information may be inconsistent.
• Fan et al. (2024) and Ren et al. (2023) are cited as related work on RAG and factual boundaries, and Schick et al. (2024) is cited for tool use. The paper does not criticize these methods.
• Toyer et al. (2024)'s Tensor Trust is cited for indirect prompt injection: hidden instructions in retrieved content may hijack model behavior.

Ethics, values, and preferences

• Chiu et al. (2025a; 2025b)'s DailyDilemmas uses everyday dilemmas to examine value preferences; Section 2.4 borrows the example of truthfully reporting versus protecting a sick child. Jin et al. (2025b), Hatemo et al. (2025), and Samway et al. (2025) are cited for trolley problems and ethical frameworks revealed by model choices.
• Pan et al. (2023)'s Machiavelli measures the trade-off between rewards and ethical behavior; Huang et al. (2025), Kang et al. (2025), and Sorensen et al. (2024) are cited for value analysis in real-world interactions or pluralistic values. Schwartz and Bardi (2001) is cited for value pluralism.
• Zheng et al. (2023) use LLMs as judges; Jiang et al. (2023; 2024), Wu et al. (2025), Zhu et al. (2024), Santurkar et al. (2023), and Kirk et al. (2024a; 2024b) are cited for individual preferences, personality, or pluralistic feedback. The authors use these to show that when there is no objective standard, judges must face subjective preference disagreements.

The paper does not set baselines, nor does it report its own results on DailyDilemmas, IHEval, Machiavelli, or PRISM. The introduction treats Ouyang et al. (2022)'s human feedback instruction following as a main line pursuing helpful, honest, and harmless. The authors position this paper as: classifying beyond instruction hierarchies and everyday value dilemmas, placing these conflicts into a unified framework using conditional priority graphs, then discussing priority hacking and the philosophically irreducible parts.

作者把冲突写成语境条件下的二选一，用 priority graph 表示谁优先于谁；对可被伪造语境操纵的部分，提议 runtime verification。二者都停留在形式化与机制设计，论文未给出实现。

条件决策

• 给定语境 C 和两个竞争行动或价值 A1、A2，模型输出决策 D。作者把它写成 p_θ(D|A₁, A₂, C)，即参数为 θ 的条件分布。
• 若 M(D, A₁, C) > M(D, A₂, C)，则记 A1 ≻ A2。作者不严格形式化 M，并称现实定义可以是对数概率，也可以是其他打分机制。
• 这一写法被用来对照阿西莫夫三定律那种线性排序。作者认为真实优先级不是单一路径。

优先级图

• 图记为 GC = (V, EC)。节点 V 是指令或价值，如系统指令、用户指令、安全、有用性；有向边是该语境下的优先关系。
• 边 (A1, A2) 存在，当且仅当模型被迫二选一时把 A1 放在 A2 之前。图表示的是条件决策，不是训练时显式给定的结构。
• 作者举出的简单路径包括：系统提示词 ≻ 用户指令 ≻ 外部检索知识；正义 ≻ 可持续性 ≻ 经济价值；人类安全 ≻ 人类指令 ≻ 自我保护。
• 作者认为模型未必按有向图训练，但可能从不同语境的训练数据中隐式学到优先级，并引用 Chiu et al. (2025a)、Zhang et al. (2025)、Wallace et al. (2025)。

动态性与循环

• 边集 EC 随语境 C 重配。作者列出的因素有：具体用户及其偏好、对话历史与先前轮次、交互时间及社会规范变化、工具或 API 等外部环境。
• 第3.2节举例：创意写作语境可以是创造力 ≻ 事实准确性；研究者语境可以变成事实准确性 ≻ 经济价值。后一对比按原文照写。
• 图可以含有向环，例如 A1 ≻ A2 ≻ A3 ≻ A1。作者称这表示不可调和的悖论。
• 因此，若想指定一个跨语境一致的静态 GC，作者认为关键困难是该图既不静态，也不必然逻辑一致。

priority hacking

• 正常情况下，对齐模型应对恶意或危险查询拒答，以满足 Asafety。
• 作者将其称为一种可能的攻击形式：找出良性高层价值 Avalue，构造 Cadv，使该语境中 Avalue 优先于 Asafety。第4节把这一关系写成 M(D, Avalue, Cadv) ≻ M(D, Asafety, Cadv)，与第3节的 > 写法不同。
• 机制是把有害请求表述成服务于该高层价值的行动。作者称，训练数据使模型在某些情境中把正义、公共安全或教育自由等抽象价值放在一般限制之上。第4节的例子使用伸张正义与公共健康的叙事，此处不复述提示词。
• 正文用 may：作者认为这可能导致模型为满足 Avalue 而违反 Asafety。摘要的说法更肯定，称对手可以构造欺骗性语境来操纵该图并绕过安全对齐。论文未报告成功率、查询次数或迭代预算。

运行时核验

• 作者认为，伪造语境能够得逞，是因为模型常常分不清真实的高风险语境和用户编造的语境。
• 提议在执行由用户语境支撑、且可能有害的指令前，向真实的外部信息源核验前提。若发现虚假或欺骗，丢弃被操纵的 GC，回到默认安全图 Gdefault。
• 第5节有两个设想，不是实验：正义叙事可检索可信新闻档案与法律数据库，查无可信报道则拒绝有害输出，作者称仍可履行 Ajustice；恶意信息注入则可向授权用户核验，并对照预定义安全协议。
• 论文未给出外部源清单、核验模型、阈值或拒答策略。作者认为，这样能使模型从单纯遵循指令，转向评估所收到的指令。

Experimental Setup

• Models tested: The paper does not report models run by the authors. GPT-4 system cards in references and models from other papers are not claimed to have been tested by the authors.
• Materials: No self-built or reproduced evaluation sets. Table 1 defines five conflict types and gives one example each. The multi-turn dialogues, retrieval contradictions, trolley problems, resource allocation, value trade-offs, and review preferences in Section 2 are examples.
• Others' data: DailyDilemmas (Chiu et al., 2025b) is only used to illustrate the dilemma between truthfulness and protection. Machiavelli, IHEval, and PRISM appear in citations; the paper does not report its own scores.
• Baselines: None set.
• Metrics and evaluation: ASR, accuracy, or human agreement are not defined; no evaluation model, threshold, sample size, or repetitions are specified.
• Judgment: Priority hacking is conceptually the model violating Asafety to satisfy Avalue. The paper does not write it as a scoreable protocol.

Main Results

The paper reports no quantitative main results. The table below transcribes Table 1's definitions and example types, not measured values.

Type	Definition (Table 1)	Example Type (Table 1)
Instruction	Two or more explicit instructions directly contradict	First asking not to name someone, then asking for the sender
Information	Parametric internal knowledge conflicts with external information	Retrieved news inconsistent with training data
Ethics	Choosing between two fundamental ethical frameworks	Utilitarianism vs. deontology in a trolley problem
Value	Two or more positive, human-aligned values conflict	Lying to avoid psychological harm to a sick child
Preference	Adjudicating between subjective, diverse, non-factual human preferences	Judging which of two poems is better

• The authors state that these five types extend from syntactic and semantic contradictions to normative and subjective conflicts, and consider them widespread in real LLM scenarios. The paper gives no occurrence counts or proportions.
• Figure 1's caption is "Five types of conflicts in current LLM applications." The converted text has no axes, legend values, or bar heights.
• Figure 2's caption is divided into three parts: priority graph, exploiting the graph to bypass jailbreak safety constraints, and communicating with external information sources to verify context. There are no numbers in the text to copy.

Conceptual Examples

• Instruction and information: Section 2.1 uses the contradiction between a persistent constraint and a direct question in a later turn, and competing goals, to illustrate that models need implicit priorities. Section 2.2 contrasts outdated parametric knowledge with retrieved documents and discusses trust issues when external documents are biased or contain hidden instructions. The paper does not report which side the model actually chose.
• Ethics, values, and preferences: The trolley problem and public resource allocation are used to illustrate that choices commit to an ethical framework; value examples include sustainability vs. economic value, and truthfulness vs. protection in DailyDilemmas. Preference examples are story or art reviews without objective standards. All are hypothetical scenarios.
• Attack and verification scenarios: Sections 4 and 5 only give mechanisms and scenarios. The authors state that fabricated contexts may lead to safety constraint bypass, and external verification can identify deceptive premises and revert to Gdefault. No before-and-after numbers.

Other Ablations and Analyses

• The paper reports no ablations, parameter sweeps, transfer evaluations, or human evaluations.
• Section 3 explicitly does not strictly formalize M, so there are no sensitivity results for thresholds or scoring functions.
• The abstract states that runtime verification enhances robustness; Section 6's tone is that it may help with factual errors or deception. Neither has experimental numbers.

Limitations and future directions noted by the authors

• Boundaries of the technical approach: Section 6 states that runtime verification may help with conflicts based on factual errors or deception, but many of the deepest dilemmas cannot be resolved this way because there is no universally accepted correct answer.
• Philosophical irreducibility: The abstract and Section 7 state that many core ethical and value conflicts are philosophically irreducible; addressing these dilemmas is a long-term open challenge for alignment.
• No ultimate ordering: Section 6 states that determining which values should be prioritized may have no ultimate answer in philosophy or human sociology. The authors also state that human values are not static and that prompting strategies can change the value hierarchy exhibited by a model; this passage does not provide specific research, models, or numbers.
• How models should act: Section 6 leaves three questions for future work: whether to refuse to answer in the face of deep ethical conflicts; whether to present arguments from different philosophical frameworks side by side; and whether to make it adjustable, allowing end users to set core value priorities before interaction.

Experimental coverage

• The content of Sections 2 through 5 is a taxonomy of five conflict types, a formalization of priority graphs, a mechanistic description of priority hacking, and a conceptual design for runtime verification.
• The paper does not report the tested models, sample sizes, number of repetitions, evaluation models, or any quantitative metrics.
• The paper does not report attack success rates, query counts, or iteration budgets for priority hacking.
• The paper does not report an implementation of runtime verification, external information sources, verification protocols, or defense evaluations.
• Table 1 provides definitions and example types for the five conflict categories, without reporting the size of each category; Section 3 does not formally define M, and does not report specific scoring mechanisms or thresholds.

The authors propose a position: the dilemma in LLM alignment may not be reducible to a single stable priority across contexts.

• Conflicts to address: Explicit instructions can contradict each other, parametric knowledge can conflict with retrieved information, consequentialism can conflict with deontology, and multiple positive values as well as different users' subjective preferences can oppose each other. The authors argue that the more autonomous models become and the more they enter social and economic scenarios, the more frequently these conflicts arise.
• Priority graph: Nodes are instructions or values; edges are priority relations in a given context, determined by the model's conditional choices, with the measurement function M not strictly specified. The graph reconfigures with users, conversation history, time and social norms, and external tools, and can contain directed cycles. The authors therefore argue that specifying a static and logically consistent global priority is difficult.
• Priority hacking and verification: An adversary can construct contexts where the model's existing high-level values override safety constraints. The authors propose verifying premises against external trusted sources before execution, and reverting to a default safe priority graph if false. The abstract states this enhances robustness; Section 6 only says it may help with factual deception. The paper has no experimental numbers.
• Authors' conclusion: Verification cannot resolve the deepest ethical and value conflicts. These conflicts reflect value pluralism, with no universal correct answer. Whether models should refuse, present different philosophical perspectives side by side, or leave core priorities to user setting, the authors consider an open long-term alignment question.