Scalable Agent Alignment via Reward Modeling: A Research Direction

Core problem: how to create agents whose behavior conforms to user intent (the agent alignment problem). Real-world tasks (ordering food, replying to emails, software engineering, running a business) do not come with scores as games do; task goals can only be obtained indirectly through user intent, so reinforcement learning is hard to apply directly.

• Why it matters: The authors believe the ultimate goal of machine learning research is to move beyond games and improve human life; to enable RL agents to complete complex tasks according to intent without hand-designed reward functions, alignment is a prerequisite.
• Shortcomings of existing methods: The authors point out that commonly used metrics such as BLEU, Inception score, and Fréchet inception distance are not aligned with user intent, and directly optimizing them produces degenerate solutions; while protocols such as demonstrations and value/advantage feedback require users to be able to do the task themselves, putting the burden of credit assignment on the user, and typically cannot achieve performance far beyond human level.
• Core assumptions: The paper relies on two assumptions—Assumption 1: user intent can be learned to sufficiently high accuracy; Assumption 2: for most tasks, evaluating outcomes is easier than producing correct behavior.
• What the paper proposes: A research direction centered on reward modeling: learn a reward function from user feedback, then optimize it with RL; apply it recursively for complex tasks (recursive reward modeling); and systematically sort out the challenges of scaling, mitigation methods, and ways to build trust.
• Nature of the goal: The authors emphasize that this is a research direction rather than a plan guaranteed to succeed, and the goal is to clarify whether reward modeling is a scalable alignment solution; the paper only discusses the technical problem of a single agent with a single user and does not address the source and aggregation of preferences.

The raising and formalization of the alignment problem

• Asimov (1942) was the earliest to echo the alignment problem through the science-fiction stories of the Three Laws of Robotics; philosophers such as Bostrom (2003, 2014) and Yudkowsky (2004) continued the discussion, and Wiener (1960) and others gave informal technical treatments.
• Dewey (2011) gave the first formal treatment, later refined by Hadfield-Menell et al. (2016) and Everitt & Hutter (2018).
• Amodei et al. (2016) discussed practical problems such as safe exploration, distribution shift, side effects, and reward hacking, and raised the problem of scalable oversight; Taylor et al. (2016) surveyed open problems in alignment.

Prototype work on learning rewards from feedback

• Christiano et al. (2017) and Kreutzer et al. (2018) used deep networks to learn from trajectory preferences; Bahdanau et al. (2018) learned from goal-state examples; Finn et al. (2016) and Ho & Ermon (2016) learned from demonstrations; Tung et al. (2018) and Ibarz et al. (2018) combined these.
• Other interaction forms: score feedback (El Asri et al., 2016), action feedback (Griffith et al., 2013), value (Knox & Stone, 2009), advantage (MacGlashan et al., 2017), trajectory preferences (Fürnkranz et al., 2012, etc.), explicit objective functions (Hadfield-Menell et al., 2017b).

Alternative alignment approaches (discussed and compared one by one in the paper)

• Imitation learning (Pomerleau, 1991, etc.): the authors hold that its data must be provided by experts and it is difficult to surpass human performance, so in the long run it is unlikely to compete with other approaches, but it can serve as a "stepping stone."
• Inverse reinforcement learning, IRL (Russell, 1998; Ng & Russell, 2000): the authors point out two flaws—the reward function cannot be uniquely identified from behavior; and it assumes humans act directly according to the reward, whereas stating preferences is often less effortful than demonstrating.
• CIRL (Hadfield-Menell et al., 2016): the authors hold that it replaces the problem of specifying a reward function with the problem of specifying a prior over reward functions, and that the optimal solution is unrealistic, but they affirm two of its insights.
• Myopic RL (TAMER, COACH): the authors hold that it shifts the burden of credit assignment to the user and limits performance, but it can be used for tasks where credit assignment is easy or as a building block of iterated amplification.
• Imitating expert reasoning (Christiano et al., 2018, etc.) and debate (Irving et al., 2018): the authors state that the former may be less likely than recursive reward modeling to compete on interpretability, and its economics are questionable; whether the core assumption of the latter (telling the truth is easier than lying) holds still requires more empirical evidence.

Theoretical agenda

• The agent foundations agenda of Soares & Fallenstein (2017) focuses on the theoretical foundations of mathematical agent models; the authors hold that even if all its problems were solved, there would still be a gap from practical alignment, and at present it should be regarded more as an intuition pump.

Positioning: the authors state that this paper, building on existing work, provides a more coherent picture and more details, integrating various AI safety problems under the single narrative thread of reward modeling.

The overall idea is to split agent alignment into two parts: learning a reward function that captures intent from user feedback ("what to learn"), then training a policy with reinforcement learning to optimize that reward ("how to do it"), called reward modeling (Figure 1).

Problem setup

• The agent interacts step by step with the environment according to a policy in a POMDP\R (a POMDP without a reward function), and separately receives intent conveyed by the user through an unspecified interaction protocol.
• The solution is a policy that produces behavior conforming to user intent; design specification problems (off-switch, side-effects, absent supervisor, containment breach, creating sub-agents, etc.) are viewed as incentive misalignment caused by an incorrect reward function.

Reward modeling

• The user trains a reward model with feedback; the reward model provides rewards to the RL agent; the two processes run in parallel (user in the loop).
• The authors say this shifts the credit assignment burden from the user to the agent, so the user only needs to evaluate outcomes; if Assumption 2 holds, teaching a reward function is easier than doing the task oneself.
• The authors give four potential advantages: no need to give feedback on every interaction; the alignment of the policy and the alignment of the reward model can be distinguished; it can directly connect to stronger RL agents; the user does not need to solve credit assignment.
• The authors give an existence argument: if the set H of histories corresponding to aligned behavior is non-empty, then there exists a reward function r such that its optimal policy produces behavior in H with probability 1; but in practice one must also consider model capacity, data volume, generalization, and whether the resulting behavior is sufficiently close to H.

Recursive reward modeling

• For tasks that are hard to evaluate directly (such as writing a novel or doing research), an agent A(k−1) trained on simpler tasks assists the user in evaluation, and then a more general A(k) is trained, recursively layer by layer (Figure 2); forms of assistance include providing auxiliary information, summarizing data, explaining internal states, and solving subproblems.
• The authors call this an instance of iterated amplification (Christiano et al., 2018) with reward modeling replacing supervised/imitation learning; and raise the open question of whether errors accumulate layer by layer and whether they can self-correct.
• The authors give an analogy to complexity theory (from Irving et al., 2018): reward modeling is like "propose—verify" in NP, and k levels of recursion correspond to k alternating quantifiers; the authors believe the general belief that P≠NP supports Assumption 2, and that the framework can cover a very general task space.

Challenges and mitigation ideas

• Five categories of challenges (Section 4): whether the feedback volume is affordable, feedback distribution shift, reward hacking (including reward gaming and reward tampering), unacceptable outcomes, and the reward-result gap (the difference between the learned reward and the reward actually optimized by the policy).
• Ten mitigation ideas (Section 5; correspondence shown in Figure 3): online feedback, off-policy feedback, using existing data, hierarchical feedback, natural language, model-based RL, side-constraints, adversarial training, uncertainty estimation, inductive bias.

Building trust

• Five ways (Section 6): design choices (such as separating the reward model from the policy, planner + executor), testing (held-out sets, adversarial evaluation), interpretability, formal verification, theoretical guarantees; the long-term goal is to produce a "safety certificate" that can convince third parties.

Experimental setup

• This is a research-direction paper and did not run new experiments of its own; the text reproduces two sets of Atari experiments from Ibarz et al. (2018) (Figure 4, Figure 5) to illustrate reward hacking risk and how reward alignment is measured.
• Setup of the reproduced experiments (per figure captions): in Atari games, learn a reward model from user preferences; Figure 4 takes the best seed of the trained reward model and freezes it, using it to train a new agent from scratch; Figure 5 compares learned rewards with true Atari rewards on 9 Atari games, averaged over 1000 timesteps.
• The paper does not report the sample size, number of repetitions (other than "best seed", the number of seeds is unspecified), algorithm details, or review method for these experiments.

Main results

The paper has no results table of its own; the contents of the two reproduced figures are as follows (only information directly given by the captions and text on the figures):

Figure	Content	Information visible on the figure
Figure 4	Three games: Hero, Montezuma's Revenge, Private Eye; training a new agent with a frozen reward model	x-axis Agent steps (millions); curves are true episode return (black) and reward model return (green)
Figure 5	Scatter plots for 9 Atari games	y-axis Learned reward, x-axis True reward; if perfectly aligned, all points should lie on a straight line

• The authors say that in Figure 4, as training proceeds the agent learns to exploit the reward model: its "perceived performance" by the reward model rises while its true performance by game score plummets.
• The authors say that in Figure 5 the reward model aligns well on games such as Beamrider, Hero, and Q*bert, and aligns poorly on games such as Private Eye, Breakout, and Montezuma's Revenge (the figure caption spells Montezuma's Revenge as Mondezuma's Revenge).
• No specific values are labeled on the figures, so the quantitative magnitude of the correlation cannot be read from the figures.

Other forms of evidence

• Existence argument: the authors give a theoretical argument—if the set H of aligned behavior histories is non-empty, then there exists a reward function r whose optimal policy produces behaviors in H with probability 1; they also note that in practice this is further limited by model capacity, data amount, and generalization ability.
• Literature evidence: the authors cite literature saying that RL from learned reward functions has succeeded on gridworlds (Bahdanau et al., 2018), Atari (Christiano et al., 2017; Ibarz et al., 2018), and continuous motion control (Ho & Ermon, 2016; Christiano et al., 2017), as examples that reward gaming may be avoidable in practice.
• Formal verification progress: the authors cite literature saying that verified models have exceeded one million parameters (Dvijotham et al., 2018b; Wong et al., 2018), and believe verifying practical-scale RL models may soon be feasible.

Limitations and future directions noted by the authors

• Success not guaranteed (Section 1 Disclaimer): The authors emphasize that success of this research direction is not guaranteed and should not be understood as a plan that will achieve alignment once executed; it merely poses research questions that can answer "whether reward modeling can scale."
• The preference-loading problem is out of scope (Section 1): Whose preferences the agent should align with, how different users' preferences should be aggregated, and when it should disobey are explicitly excluded from the paper's scope.
• Assumption 2 has counterexamples (Section 1): Tasks with low-dimensional outcome spaces (such as yes/no questions) do not satisfy "evaluation is easier than behavior" unless the user also requires explanations.
• Reward gaming is the biggest open problem (Section 4.3): The authors say whether reward gaming can be overcome in practice is "arguably one of the biggest open problems, and possibly the biggest weakness of reward modeling."
• Error accumulation unknown (Section 3.2): Whether errors of a narrower agent A(k−1) are amplified when training A(k), and whether the training process can self-correct, are important open problems.
• Alternative routes such as debate await validation (Section 7.6): Whether the core assumptions of debate hold outside easily checkable domains, and whether humans can judge the more persuasive debater, require more empirical evidence.

Scope of experimental coverage

• The paper did not run its own models or experiments; the only quantitative content is the reproduction of Figure 4 (3 Atari games) and Figure 5 (9 Atari games) from Ibarz et al. (2018).
• The paper does not report the sample size, number of seeds, algorithm hyperparameters, or statistical tests of these reproduction experiments.
• The paper proposes no new benchmarks, datasets, or evaluation protocols; Section 8 proposes that subsequent experiments could use existing RL benchmarks with reward functions by hiding the reward and synthesizing user feedback, which is a plan rather than completed work.
• The authors state that the challenge list (Section 4) is not intended to be exhaustive and only aims to cover the most important parts.

This is a 2018 research direction paper by DeepMind authors, centered on the agent alignment problem—how to create agents whose behavior conforms to user intent—and gives a technical route with reward modeling at its core.

• Problem: Real tasks have no ready-made reward function; the goal is only implicit in user intent. Commonly used proxy metrics (BLEU, Inception score, etc.) are misaligned with intent, and directly optimizing them produces degenerate solutions.
• Method highlights: Alignment is split into two steps: "learning a reward model from user feedback" and "optimizing that reward with RL"; for complex tasks it is applied recursively, letting a trained agent assist the user in evaluating the next generation of agents (recursive reward modeling, which the authors call an instance of iterated amplification). The route relies on two assumptions: user intent can be learned accurately enough; and evaluating outcomes is easier than producing behavior.
• Challenges and countermeasures: The authors sort out five categories of challenges (feedback volume, distribution shift, reward hacking, unacceptable outcomes, reward-result gap) and ten mitigation ideas (online/off-policy feedback, natural language, model-based RL, side-constraints, adversarial training, uncertainty estimation, etc.; the correspondence is shown in Figure 3), and propose five ways to build trust (design choices, testing, interpretability, formal verification, theoretical guarantees).
• Evidence: The paper has no experiments of its own; reproducing Ibarz et al. (2018) shows that a frozen reward model is exploited by a new agent and true return declines (Figure 4), and that across 9 Atari games the degree of alignment between the learned reward and the true reward varies (Figure 5).
• Authors' conclusion: The authors consider this direction "shovel-ready" and immediately amenable to empirical research, but explicitly state that its success is not guaranteed; whether reward gaming can be overcome in practice is called one of the biggest open problems; the source of preferences and the aggregation problem are outside the paper's scope.