Prover-verifier games improve the readability of LLM outputs
Prover-Verifier Games improve legibility of LLM outputs
Quick read
Based on the paper's full text (AI-generated); refer to the original
The OpenAI Superalignment team proposes checkability training based on a prover-verifier game, which simultaneously improves the correctness of model solutions and the accuracy of human verification on elementary school math problems. Methodologically, a small verifier and provers in two roles are trained alternately: the helpful prover is rewarded for giving correct answers that the verifier accepts, and the sneaky prover is rewarded for giving incorrect answers that fool the verifier; the provers are optimized with PPO. Results show that as training rounds progress, the verifier becomes harder to break against various sneaky-only attacks, the sneaky prover's errors become increasingly locally subtle, and the helpful prover's pass rate rises together with the judgment accuracy of time-limited human contractors. The authors note that reinforcement learning based only on answer correctness yields solutions that are correct but hard to understand, and that checkability training still requires some performance cost.
- Large language model outputs need clear, checkable reasoning (which the authors call legibility) for people to trust them; but optimizing chain-of-thought solely by answer correctness may make solutions hard to check. The authors study how to balance accuracy and checkability on elementary school math problems.
- The authors propose checkability training inspired by the Prover-Verifier Game: iteratively train a small verifier to judge the correctness of solutions, while using RL to train a "helpful" prover to produce correct and accepted solutions and a "sneaky" prover to produce incorrect solutions that can fool the verifier; and they prove that the verifier-led Stackelberg equilibrium is equivalent to completeness and soundness.
- Optimizing only for correctness can yield high-accuracy solutions that are hard for humans to evaluate; checkability training strikes a trade-off between accuracy and checkability (there is a legibility tax). As rounds increase, the verifier becomes more robust to various sneaky attacks, the flaws in sneaky solutions become more concealed and harder for humans to identify, and both the correctness rate of helpful solutions and the accuracy of human judgments rise.
- The study is limited to GSM math problems that have standard answers and can be explained in natural language; directly training CoT into checkable solutions may limit accuracy, and the authors suggest separating solving from explanation; unlabeled or cross-domain generalization was not studied, verifiers that are very small or the same size as the prover perform poorly, and some human experiments are pilots with design flaws.
Presents a checkability training scheme based on a prover-verifier game on elementary school math tasks, and reports that the judgment difficulties of small-model verifiers can transfer to human evaluators.