Enforcing axioms for AI alignment under loss-based rules

Core question: reward models trained by loss-based rules (such as the Bradley–Terry loss in RLHF) violate Pareto optimality (PO); can this violation be eliminated through stronger reward models or better comparison data.

• Problem setting: constitutional-style alignment treats each principle as a voter, comparing two responses according to that principle (e.g., more helpful, less toxic). The authors state that if response A is better than B on all principles, A should receive a higher reward; this is PO (i.e., Unanimity).
• Existing finding: Ge et al. (2024) proved in a linear social choice model that the optimal linear reward may violate PO—all voters prefer A, yet B receives a higher reward; their counterexample uses only two weighted principles and covers the HHH setting. The authors consider this result counterintuitive (the paper's original wording: "Finding this result counterintuitive").
• Core assumption: principles are modeled as linear directions in a fixed embedding space, a simplifying assumption inspired by the Linear Representation Hypothesis; the authors argue that the linear assumption and axioms such as PO are more realistic in constitutional-style alignment with a small number of principles than in general RLHF.
• What the paper does: (1) prove that PO and PMC can still fail for polynomial rewards of any fixed degree and for single-hidden-layer ReLU MLP rewards of any fixed width d≥2; (2) generalize PO from a fixed candidate set to all comparison directions over the entire embedding space; (3) under the idealized setting with uniform comparison directions and equal comparison lengths, restore PO for linear rewards, and note that this motivates restoring axiomatic guarantees through dataset design.

Correspondence between social choice and alignment

• Christiano et al. (2017) proposed RLHF, and Munos et al. (2024) proposed NLHF; the paper notes that the ranking induced by the optimal solution of the Bradley–Terry loss in RLHF is equivalent to the Borda ranking (Anderson et al., 2009; Siththaranjan et al., 2024), and that NLHF coincides with the von Neumann winner of Dudík et al. (2015), and thus with the maximal lotteries of Fishburn (1984).
• Position papers such as Conitzer et al. (2024) and Dai and Fleisig (2024) advocate viewing RLHF and alignment from a social choice perspective.

Linear social choice model (the most direct prior work)

• Ge et al. (2024) proposed the linear social choice model: voters are linear directions on a fixed embedding; they analyze the ranking induced by the optimal linear reward on a fixed candidate set, find that PO and PMC can be violated, and adapt a combinatorial social choice rule to obtain guarantees. This paper builds on this to study the robustness of axiom violations under loss-based rules.
• Noothigattu et al. (2020) study the axiomatic properties of reward functions as MLEs of random utility models, which is the basis of Ge et al.
• Elkind et al. (2022), on restricted preference domains, is cited by the paper as the closest social choice counterpart to "linear preferences on a fixed embedding".

Other work at the intersection of alignment and social choice

• Procaccia et al. (2025) analyze clone robustness and state that appropriate reweighting of the Bradley–Terry loss satisfies their axioms.
• Qiu (2024)'s representative social choice, Gölz et al. (2025)'s extension of the distortion framework, Xiao et al. (2025)'s analysis of single-annotator RLHF, and Kim et al. (2025)'s proportional alignment proposal; the paper notes that these works mostly adopt the classical social choice setting with unstructured alternatives, which differs from this paper's metric (embedding space) setting.

Constitutional-style alignment practice (Appendix A)

• Bai et al. (2022a) and Glaese et al. (2022) proposed principle-guided supervision (judging by explicit criteria such as HHH); Bai et al. (2022b)'s Constitutional AI uses written principles to guide binary comparisons; Cui et al. (2023), Sun et al. (2023), and others implement similar ideas under different names.

Positioning of this paper: Ge et al. restore axioms with a combinatorial social choice rule, whereas this paper focuses on loss-based rules compatible with modern ML pipelines, examining the two paths of increasing reward-model capacity and improving comparison data.

Overall approach: within Ge et al.'s (2024) linear social choice framework, model principles as linear directions (voters) in an embedding space, and study whether loss-based voting rules can satisfy the two axioms PO and PMC; first give impossibility results (polynomial, MLP rewards), then a possibility result under uniform data (linear reward).

Problem Setup and Formalization

• Alternatives and voters: alternatives are points in R^d; each of n voters provides a set of pairwise (strict) comparisons Pi, where (a,b)∈Pi means voter i prefers b over a; the voting rule outputs a reward function r whose induced ranking aggregates voter preferences.
• Loss-based rule: given a loss function ℓ, the total loss is L(r) := ∑_i∈[n]∑_{(a,b)∈ Pi} ℓ(r(a)−r(b)), and the rule outputs the reward function minimizing L; taking ℓ as cross-entropy yields the Bradley–Terry loss used in RLHF.
• Axioms: PO requires that comparisons (a,b) on which all voters agree are correctly ordered by r (r(a)<r(b)）；PMC 要求当存在与严格多数意见一致的排序时，r 诱导该排序。

PO 违反的直觉（Section 3）

• 简化最小例：单个投票者 v=(ε,1)、三个候选、奖励方向 θ 受单位范数约束时，最优线性奖励违反 PO；作者借此说明范数约束如何导致 PO 失败。
• 机制解释：多个投票者意见不一致时，损失同时包含沿两个相反比较方向的惩罚项；当这些方向张成嵌入空间时，其合效果相当于隐式范数约束。作者把范数约束解释为有限"预算"：更常见方向或差向量更长的比较对预算的使用影响更大。

不可能性结果（Section 4）

• 多项式奖励（Theorem 1）：对任何严格凸、有下界、可微且 ℓ′(0)>0, including Bradley–Terry, construct an instance on R^2 with m+1=d(d+1)+2 candidates and two weighted voters (α∈(1/2,1); taking α=2/3 gives three unweighted voters), such that any polynomial reward of fixed degree d violates PO and PMC; the construction places candidates on lines Lj (slope −2) so that the unique optimal reward for the degenerate instance is realized by the linear polynomial p*(x,y)=−x−y, then uses Berge's maximum theorem to show that after a small perturbation the optimal solution remains close to p*.
• MLP rewards (Theorem 3): for any fixed width d≥2, there exists a finite parameter bound W such that a single-hidden-layer ReLU MLP reward still violates PO; the construction uses m+1=O(d^3) candidates and a piecewise affine function z (slopes −1,−M,…,−M^{d−1}), and uses the pigeonhole principle (3q candidates per block, where q=1+d+d^2 is an upper bound on the number of regions in the hinge-line partition) to force any optimal MLP to locally agree with z near each block, then takes M avoiding a finite set Ed to force all hinge lines to be parallel, and finally completes the perturbation argument likewise with Berge's theorem.

Restoring PO under Uniform Data (Section 5)

• Directional formulation: a linear voter's preference over a comparison (a,b) depends only on the direction of the difference vector b−a, and the length determines the scale at which it enters the loss; thus comparison data can be described by a measure µ over directions and a length function R, and PO is generalized to all of S^{d−1} (Definition 3), covering directions not appearing in the training data.
• Idealized setting and positive result: when comparison directions are uniformly distributed and all lengths are 1, the total loss is L(θ) := ∑_i=1ⁿ ∫_{x∈ S^d−1, ⟨ vi,x⟩≥ 0} ℓ(−⟨θ,x⟩) dx; Theorem 4 proves that as long as at least two voters have different directions, any convex, non-decreasing, lower-bounded, differentiable loss-based rule with ℓ′(0)>0 satisfies PO under this setting (the core of the proof: under the same norm, θ closer to the voter direction has lower loss, i.e., Claim 7).
• Two negative results: Proposition 1 states that when directions are uniform but lengths vary according to an elliptical radial function, the BT loss still violates PO (two dimensions, two weighted voters); Theorem 5 states that even under the idealized uniform setting, strictly convex losses still violate PMC. The authors consider whether PMC is even desirable under this setting to be unclear, since it is a discontinuous requirement when the majority proportion crosses 0.5.

Experimental Setup

• The paper contains no experiments: no tested models, datasets, baselines, metrics, or review methods; all results are mathematical theorems and proofs.

Main Results

The paper's main results are the following theorems (all are propositions proved by the authors, not experimental findings):

Result	Setting	Conclusion (as stated by the authors)
Theorem 1	Strictly convex, lower-bounded, differentiable loss with ℓ′(0)>0; R^2; three voters	Any bounded-degree polynomial reward still violates PO and PMC
Theorem 3	Same loss as above; R^2; three voters	Any single-hidden-layer ReLU MLP of fixed width d≥2 (parameter bound [−W,W], W finite) still violates PO
Proposition 1	BT loss; two dimensions; two weighted voters; directions uniform, lengths given by an elliptical radial function	Still violates PO when directions are uniform but lengths are unequal
Theorem 4	Convex, non-decreasing, lower-bounded, differentiable loss with ℓ′(0)>0; directions uniform, lengths equal; at least two voters with different directions	Linear reward satisfies PO (for directions over all of R^d)
Theorem 5	Strictly convex, non-decreasing, lower-bounded, differentiable loss with ℓ′(0)>0; idealized uniform setting	Still violates PMC

• The authors state that Theorem 1 and Theorem 3 show that increasing reward-model expressiveness (polynomials, universal-approximating MLPs) cannot restore PO.
• The authors state that Theorem 4 shows that under idealized data with uniform directions and equal lengths, loss-based training of a linear reward can restore PO, which motivates restoring axiomatic guarantees through data curation and reweighting.

Construction and Proof Highlights

• Instance for Theorem 1: two weighted voters v1=(1,0) (weight α∈(1/2,1)) and v2=(0,1), which agree only on the comparison between c0 and c1; candidates are placed on d lines of slope −2, with d+1 points on each line, so that the optimal reward for the degenerate instance is uniquely realized by p*(x,y)=−x−y (Figure 1 shows a schematic of candidate positions for d=2).
• Instance for Theorem 3: m+1=O(d^3) candidates are divided into d blocks of k=3q each (q=1+d+d^2), placed on the affine segments of a piecewise affine function z; using Zaslavsky's region count (Fact 1) and the pigeonhole principle, any optimal width-d MLP is forced to agree with the local affine piece of z near each block.
• Proof of Theorem 4: the core is Claim 7 (under the same norm, θ with a larger inner product with the voter direction has strictly lower loss), together with existence of a minimum (Claim 8) and θ=0 not being optimal (Claim 9); Figure 2 gives the geometric intuition: if the half-space Hθ does not contain the voter cone P, it can be rotated to a supporting position to reduce the loss.

Other Ablations and Analysis

• The minimal single-voter, three-candidate example in Section 3: under a norm constraint, the optimal linear reward violates PO, used to explain how multi-voter disagreement produces an implicit norm constraint.
• Proof instance for Theorem 5: in R^2, v1=(0,1), v2=(1,0), ratio p∈[1/4,1/2]; at p=1/2 the unique minimum solution θ* satisfies θ1*=θ2*>0, and by continuity at p=1/2−δ the minimum solution is still not a multiple of the majority direction v2.
• The paper notes that the conclusion of Theorem 4 also holds for "a common bounded positive length distribution independent of direction and voters" (footnote 7).

Limitations and Future Directions Noted by the Authors

• Extending the impossibility: the authors conjecture that the impossibility results can be extended to MLPs of any fixed depth and width (on frozen embeddings), and then to fixed-architecture transformers, under the same loss assumptions and appropriate finite parameter bounds, but this paper does not prove it (Section 6).
• Deriving sampling bounds: the authors state that a natural next step is to derive explicit sampling bounds under the uniform setting (Section 6).
• Formalizing reweighting and data query strategies: the authors propose a research direction—whether data can be collected more targeted (more efficiently) to achieve PO (Section 6).
• Generalizing other axioms: the authors state that future research should consider how other axioms extend to the embedding space and are satisfied through data selection (Section 6).
• Dependence of axiomatic guarantees on the data distribution: the authors raise an open question—to what extent guarantees should depend on the frequency with which comparisons appear, and whether there is a meaningful reference measure on the embedding space for evaluating or rebalancing data (Section 6).
• Cost of length normalization and direction coverage: the authors state that comparison lengths can be balanced through loss normalization, whereas more uniform direction coverage may require reweighting comparisons and, in the worst case, collecting additional data (Section 6).

Scope of Experimental Coverage

• The paper is purely theoretical work with no experiments; results cover: bounded-degree polynomial rewards (Theorem 1), single-hidden-layer fixed-width ReLU MLPs (Theorem 3), and linear rewards under uniform and equal-length data (Theorem 4).
• The impossibility instances are all constructed in two dimensions (R^2) with three voters (or two weighted voters) (Theorems 1 and 3; Proposition 1 uses two weighted voters).
• The positive result (Theorem 4) requires comparison directions to be uniformly distributed, comparison lengths to be equal (or a common bounded distribution independent of direction and voters, footnote 7), and at least two voters with different directions.
• The analysis is based on the linear social choice model: voters are linear directions in a fixed embedding space, which the authors state is a simplifying assumption inspired by the Linear Representation Hypothesis (Section 1).
• The paper reports no validation on real language models, real preference data, or actual reward-model training.

An alignment theory paper from a social choice perspective, studying whether loss-based reward training in constitutional-style alignment can satisfy Pareto optimality (PO).

• Problem: principles are treated as linear directions (voters) in an embedding space; Ge et al. (2024) already proved that the optimal linear reward can violate PO—all principles prefer A, yet B receives a higher reward. The authors ask how robust this axiom violation is.
• Method: prove theorems within the linear social choice framework; first give intuition with a minimal single-voter example (voter disagreement produces an implicit norm constraint, akin to a finite "budget", where common directions and long difference vectors have greater influence), then take two paths: increasing reward-model capacity and improving comparison data.
• Main results (as stated by the authors):
  • Theorem 1: any bounded-degree polynomial reward still violates PO and PMC (two dimensions, three voters).
  • Theorem 3: any single-hidden-layer ReLU MLP of fixed width d≥2 still violates PO under a finite parameter bound.
  • Theorem 4: under the idealized setting with uniform comparison directions and equal lengths, a linear reward satisfies PO (and PO is generalized to directions over the entire embedding space, covering unseen comparisons).
  • Proposition 1 and Theorem 5: PO still fails when directions are uniform but lengths are unequal; PMC still fails under the uniform setting.
• Authors' conclusions and implications: increasing reward-model expressiveness cannot restore PO, whereas controlling the comparison data distribution can (under the idealized setting); the authors argue this motivates restoring axiomatic guarantees through data curation, reweighting, and normalizing comparison lengths while retaining loss-based training, and propose future directions such as deriving sampling bounds, formalizing data query strategies, and extending the impossibility to deeper networks.