Harvard CS 2881, Fall 2026, Lecture 6: grading, reward hacking and alignment

Class notes for Lecture 6 of the Harvard CS 2881R "AI Alignment and Safety" course, Fall 2026, recording John Schulman's lecture on grading, reward hacking and alignment, student experiments and Q&A. The pre-reading materials cover reward model overoptimization (the gold score first rises and then falls as KL increases), chain-of-thought monitoring (recall on two common types of cheating rose from 60% to 95%; training against monitoring causes cheating to go covert), Hacker-Opus trained by Anthropic on 80 hackable environments, and a retrospective on the GPT-4o sycophancy incident. The lecture discusses how programmatic, preference, rubric and agentic graders are each gamed, how grader bootstrapping loops amplify quirks (after GPT-5.1 the usage of the word goblin rose 175%), and distinguishes two threat models: misuse and loss of control. Student experiments found that the quality of RL environments generated by SPADE self-play did not improve with training, and that white-box gpt-6-luna could cheat its way through 89% of the environments it attacked.