Study: Pareto optimality of AI alignment under loss training

1. 07:44
   arXiv: Alignment, Deception & Oversight

   Enforcing axioms for AI alignment under loss-based rules

   Alexandros Hollender and Sonja Kraiczy study in an arXiv paper whether methods such as RLHF that train reward models by minimizing the loss on pairwise comparisons of responses can satisfy the Pareto optimality (PO) axiom. Inspired by Constitutional AI, they treat each principle as a voter comparing responses according to that principle, and represent it as a linear direction in a fixed embedding space. In this linear social choice model, Ge et al. (2024) have shown that optimal linear reward models may violate PO, i.e., one response can be less helpful and more toxic than another yet receive a higher reward. The paper further shows that reward models that are polynomials of any fixed degree, or single-hidden-layer ReLU MLPs of any fixed width d≥2 with suitably finite bounds on parameters, may still violate PO; whereas under the idealized assumptions of uniform comparison directions and equal comparison lengths, loss-based training of linear rewards can recover PO. The authors accordingly propose that dataset design is one path to restoring PO in constitutional-style alignment while retaining loss-based training.

Current attention 27·Peak in the comparable period 30 (Nov 11 14:00)·Change over the comparable past 24 hours -4%