OpenAI studies weak-to-strong generalization: eliciting strong capabilities with weak supervision

核心问题：弱监督者（如人类）能否监督远比自己强的模型，并激发其全部能力？ 作者将这一超级对齐（superalignment）难题转化为可实证的类比：用弱模型生成的标签微调强预训练模型，观察其泛化。

• 问题场景：RLHF 依赖人类评估模型行为，但作者认为未来超人模型的复杂行为（如百万行代码是否安全、是否忠实遵循意图）人类无法可靠评估，朴素人类监督如何泛化到这些行为是未知的。
• 现有研究的不足：作者称以往对齐研究要么直面核心挑战但限于理论框架和玩具问题，要么实证研究人类监督当今模型、未触及超人模型带来的核心挑战，缺少既能捕捉核心挑战又能迭代实证的设置。
• 核心假设：强预训练模型对对齐相关任务已有良好表示，弱监督者不需要教它新能力，只需激发它已有的知识，因此强模型可能泛化超出弱监督水平，作者称之为 weak-to-strong generalization。
• 论文的贡献：提出 weak-to-strong learning 实证设置，在 NLP、chess、reward modeling 三类任务上系统测量，并测试辅助置信损失、bootstrapping、生成式微调等改进方法。
• 设置定位：作者称该设置去除了以往工作的一个主要差异，但并非最终完美类比的设置；剩余差异（imitation saliency、pretraining leakage）在 Section 6 讨论。

Weak supervision and noisy-label learning

• Weak-to-strong learning is a kind of weak supervision; related literature includes learning from unreliable labels (Bach et al., 2017; Ratner et al., 2017) and noisy-label learning (Song et al., 2022), with common methods such as bootstrapping, noise-robust losses, and noise modeling.
• The paper notes the difference: the weak supervision errors here are instance-dependent and harder to handle than uniform label noise; it also relates to semi-supervised learning, and Appendix C studies an easy-to-hard generalization variant.

Teacher-student training and knowledge distillation

• The framework of first training a teacher and then training a student on pseudo-labels is widely used in semi-supervised learning, domain adaptation, and knowledge distillation (Hinton et al., 2015; Xie et al., 2020, etc.).
• The paper notes the difference: this work focuses on the setting where the student is far stronger than the teacher; although Furlanello et al. (2018) and Xie et al. (2020) consider students no weaker than teachers, their students are randomly initialized and have access to ground-truth labels. The confidence auxiliary loss here is similar to the conditional entropy minimization of Grandvalet & Bengio (2004).

Pretraining robustness, debiasing, and alignment methods

• Pretraining yields more robust representations, and fine-tuning often harms out-of-distribution generalization; mitigation methods include weight ensembling and fine-tuning only some layers. The authors did not obtain strong results in preliminary exploration (Appendix B).
• Debiasing literature mostly handles known biases; this work can be viewed as a debiasing problem with unknown bias. Imitation learning and preference learning (RLHF, Constitutional AI) both assume high-quality human supervision, which the authors consider of unknown effectiveness for superhuman models.

Scalable oversight and eliciting latent knowledge

• Scalable oversight improves human supervision through debate, critique, and task decomposition (Irving et al., 2018; Leike et al., 2018); this setting resembles the sandwiching proposal of using strong and weak humans (Cotra, 2021; Bowman, 2022).
• ELK (Christiano et al., 2022) aims to elicit latent knowledge under worst-case assumptions; the authors consider this setting a general method for empirically studying ELK and honesty.

Positioning: the authors state that this work runs counter to most teacher-student training work, focusing on the setting where the student is far stronger than the teacher and supervision quality is weak, with the goal of generalizing beyond human supervision rather than improving human supervision itself.

Overall approach: replace "humans supervising superhuman models" with "weak models supervising strong models," and empirically measure this using GPT-4-series pretrained base models (spanning 7 orders of magnitude of pretraining compute) in three settings: NLP, chess, and reward modeling.

Problem setup and core metric

• Three-step procedure (Section 3): 1) Fine-tune a small pretrained model with ground-truth labels to obtain a weak supervisor, whose predictions on a held-out set serve as weak labels; 2) fine-tune a strong model with the weak labels to obtain a strong student, whose performance is weak-to-strong performance; 3) fine-tune the same strong model with ground-truth labels as strong ceiling performance.
• Core metric: PGR = (weak-to-strong − weak)/(strong ceiling − weak), measuring the fraction of the performance gap recovered by weak supervision; PGR is 1 for perfect generalization and 0 when not exceeding the weak supervisor.

Three task settings

• NLP: 22 popular classification datasets, uniformly converted to class-balanced binary classification, with weak labels as soft labels (class probabilities); fine-tune for 2 epochs with batch size 32, early stopping on weak-label validation accuracy.
• Chess puzzles: predict the optimal first move for a position, a generative task; weak labels obtained by sampling from the weak model at temperature 0; train for 5 epochs without early stopping; 50k+50k puzzles for training and 5k for testing.
• ChatGPT reward modeling: use the internal comparison data from training the ChatGPT RM to predict pairwise comparison outcomes; train for 1 epoch with batch size 220 without early stopping; only RM accuracy is studied, not RL steps.

Improvement methods

• Bootstrapping (Section 4.3.1): construct an increasing model sequence M1→M2→…→Mn, where each step fine-tunes the next model with new weak labels generated by the previous model; each bootstrapping round runs weak-to-strong learning three times (two intermediate sizes), evaluated on chess.
• Auxiliary confidence loss (Section 4.3.2): L_conf(f) = (1−α)· CE(f(x), f_w(x)) + α· CE(f(x), f̂_t(x)), i.e., adding to the weak-label cross-entropy a term reinforcing the strong model's own predictions (hardened by threshold t); t is adaptively chosen so that exactly half the samples in a batch exceed the threshold, αmax is 0.75 for the largest student and 0.5 for the others, with linear warmup over the first 20% of training. The design motivation is to let the student learn the supervisor's intent without imitating its errors.
• Generative fine-tuning (Section 5.2.2): perform unsupervised fine-tuning with a language modeling objective on all prefix-completion pairs in the RM data (ignoring human preferences) to increase the task's salience in the strong model; the authors note that completions may come from humans or stronger models, and there may be leakage similar to pretraining leakage.

Analysis and evaluation methods

• Imitation analysis (Section 5.1): measure overfitting to weak labels using ground-truth test accuracy during training; directly measure imitation using student-supervisor agreement (the fraction of test samples on which the student and weak supervisor predictions agree).
• Salience analysis (Section 5.2): compare zero-shot, 5-shot prompting, and fine-tuning; use linear probes (logistic regression) to measure the degree of linear representation of the task in final-layer activations.
• Hyperparameter tuning: tune only on small models and a subset of 8 datasets; GPT-4 and about half of the NLP tasks remain held out throughout the project.

Experimental Setup

• Models tested: pretrained base models of the GPT-4 series, spanning 7 orders of magnitude (OOM) of pretraining compute; a footnote states that this series shares the architecture and pretraining data of GPT-4 but does not include GPT-2, GPT-3, or GPT-3.5, and the main text describes compute tiers as "GPT-2-level" and "GPT-3.5-level".
• Datasets: 22 NLP binary classification datasets (Table 1); chess puzzles (Schwarzschild et al., 2021b, 50k+50k training, 5k test); an internal ChatGPT reward modeling dataset (size not stated).
• Baselines and ceiling: weak supervisor performance (weak), naive weak-label fine-tuning (naive baseline), ground-truth fine-tuning ceiling (strong ceiling); improved methods are bootstrapping, auxiliary confidence loss, and generative fine-tuning.
• Metrics: test accuracy and PGR; NLP results are the median over 22 tasks; NLP random guessing is 50%.
• Repeats and error: most curves do not report the number of repeats; Figure 13(b) reports s.e.m. error bars; Figure 27 reports mean±standard deviation.

Main Results

Main results are reported as curves (Figure 3); the paper gives no per-point numerical table. The table below summarizes representative numbers given in the main text.

Swipe horizontally to read the table

Setting	Naive fine-tuning PGR	Best method	Best method PGR	Source
NLP (GPT-2-level supervising GPT-4)	about 25% (smallest weak supervisor with largest student)	auxiliary confidence loss	near 80%	Section 4.3.2
NLP (largest student overall)	often above 50%	—	—	Section 4.2
Chess (small supervisor-student gap)	can exceed 40%	bootstrapping	improvement, especially for large students	Section 4.2, 4.3.1
ChatGPT RM	about 10%, almost never above 20%	generative fine-tuning	improvement of about 10–20%	Section 4.2, 5.2.2

• The authors state that in all three settings the strong student almost universally exceeds the weak supervisor, and that on NLP PGR rises as weak supervisor size and student size increase.
• The authors state that on chess PGR is near zero with the smallest weak model, and that for a given weak supervisor PGR decreases as the student grows, with a concave curve and possible inverse scaling.
• The authors state that on RM naive weak-to-strong generalization is poor, which they take as empirical evidence that naive RLHF may be hard to scale to superhuman models.

Overfitting and Imitation Analysis (Section 5.1)

• What was measured: the strong student's ground-truth test accuracy during training, and student-supervisor agreement.
• Results: on the RM task, when the weak-strong gap is large, weak-to-strong accuracy first rises then falls within less than one epoch; ground-truth early stopping (the "cheating" method) gives about a 5 percentage point PGR improvement on RM, and on NLP is 15 percentage points above the end-of-training model and 10 percentage points above early stopping on weak labels. Under naive fine-tuning, student-supervisor agreement is often higher than weak supervisor accuracy; larger students have lower agreement with supervisor errors (inverse scaling).
• Authors' interpretation: the student imitates some of the supervisor's errors; confidence loss reduces agreement with supervisor errors (Figure 8c), and in some settings agreement is even lower than the student's ground-truth accuracy.

Prompting and Saliency Experiments (Section 5.2)

• Few-shot comparison: on 7 representative NLP tasks, zero-shot vs. 5-shot is worse for small models and greatly improved for large models, and on the largest model 5-shot is comparable to fine-tuning; 5-shot with weak labels is worse than 5-shot with ground-truth labels, and on the largest student 5-shot weak-to-strong performance can be comparable to or better than naive fine-tuning, but still worse than fine-tuning with confidence loss (Figure 9).
• Generative fine-tuning (RM): after unsupervised language-modeling fine-tuning on RM data, even with the ceiling given the same fine-tuning, PGR still improves by about 10–20%; combined with ground-truth early stopping it reaches about 30–40% PGR (Figure 10, 17).
• Linear probe (NLP subset): ground-truth linear probe averages 72%, ground-truth fine-tuning 82%; weak-label fine-tuning followed by a ground-truth linear probe reaches 78%, closing 60% of the gap between linear probe and fine-tuning, and outperforms naive weak-to-strong fine-tuning (Figure 11).

Other Ablations and Analyses

• Scope of bootstrapping: clear improvement on chess; on NLP preliminary experiments show only a small improvement, and on RM no improvement (Section 4.3.1).
• Confidence threshold filtering: training only on the 5%–20% of samples the weak supervisor is most confident about helps only when weak labels are very poor (about 60% accuracy), and stops helping once weak labels reach 70%–80% (Appendix B).
• Ineffective methods: LP-FT, weight decay/dropout, LoRA, data-augmentation consistency loss, generalized cross-entropy, label noise addition, and weight averaging brought no improvement (Appendix B).
• Easy-to-hard (Appendix C): on chess, training only on easy puzzles with Elo≤800 still gives good average test performance but is comparable to or slightly worse than zero-shot on hard puzzles (Elo≥2000); on NLP, raising the difficulty cutoff generally improves accuracy.
• ImageNet (Appendix D.1): AlexNet (56.6% top-1) supervises DINO ResNet-50/ViT-B/8 linear probes; students reach 60.7%/64.2%, PGR 57.8%/41.5% (Table 3).
• Structure of weak-label errors (Appendix E): at the same weak-label accuracy, a perfectly simulable error structure generalizes almost not at all, while random noise labels generalize best; putting weak labels directly into the prompt, making imitation trivial, lowers average PGR to 0.002±0.003 (baseline) and 0.046±0.108 (confidence loss) (Figure 27).

Limitations and future directions noted by the authors

• Methods are not consistently effective (Introduction, Section 4.3): the authors state that no method works in all settings, that the RM setting is still far from recovering the full performance gap, and that the methods are more a proof of concept than a deployable practical solution.
• Imitation saliency discrepancy (Sections 3, 6.1): future superhuman models may easily imitate human errors, whereas the strong models in this paper were not explicitly pretrained to imitate weak models, so "imitating the weak supervisor" may be easier to avoid in this setting than in the future, potentially inflating today's generalization performance.
• Pretraining leakage discrepancy (Sections 3, 6.1): pretraining data implicitly contains human supervision, and many tasks may have been indirectly observed during pretraining, so eliciting capabilities may be artificially easier; superhuman knowledge may be more latent and harder to elicit, and the authors speculate that this inflates prompting methods more than fine-tuning.
• Future direction—a more analogous setting (Section 6.2.1): fix the remaining discrepancies, verify with e.g. a third-grade-level human supervising the strongest model that the discrepancies are not severe, extend to complex generative tasks, and test the robustness of weak-to-strong classifiers with high PGR under optimization pressure (e.g., RL).
• Future direction—scalable methods and scientific understanding (Sections 6.2.2, 6.2.3): find unsupervised properties that specify desired generalization without ground truth; answer why NLP and RM results differ, what makes concepts easy to elicit, whether generalization error can be estimated without labels, and whether it can be extrapolated with scaling laws.

Scope of experiments

• The models tested are GPT-4-series pretrained base models (spanning 7 orders of magnitude of compute); the appendix additionally tests AlexNet and two DINO vision models; the paper does not run GPT-2/3/3.5 themselves in the experiments.
• Tasks cover 22 NLP binary classification datasets, chess puzzles, and the ChatGPT RM internal dataset; the appendix covers difficulty analyses on ImageNet and MATH; the RM experiments measure only reward model accuracy and perform no RL optimization steps.
• Evaluations of improvement methods include bootstrapping (mainly on chess), auxiliary confidence loss (mainly on NLP), and generative fine-tuning (mainly on RM); Appendix B reports methods that did not work, such as confidence thresholds, LP-FT, LoRA, and weight averaging.
• The paper does not report the number of random seeds or repetitions for most experiments; apart from the s.e.m. in Figure 13(b) and the standard deviation in Figure 27, no error bars or statistical tests are given.
• Hyperparameters are tuned only on small models and a subset of 8 NLP datasets, with GPT-4 and about half of the NLP tasks held out throughout (Section 3 footnote, Appendix A.1).

An empirical alignment study: it converts "humans supervising superhuman models" into "weak models supervising strong models," measuring weak-to-strong generalization with GPT-4-series pretrained base models in three settings: NLP, chess, and ChatGPT reward modeling.

• Problem addressed: RLHF relies on reliable human evaluation, but the authors argue that humans cannot reliably supervise the complex behavior of superhuman models; an analogous setting that can be iterated on empirically today is needed.
• Method highlights: fine-tune a small model with ground truth to obtain a weak supervisor → fine-tune a strong student with weak labels → fine-tune a strong model with ground truth as the ceiling, using PGR to measure the recovered performance gap; improvements include auxiliary confidence loss, bootstrapping, and generative fine-tuning.
• Naive fine-tuning is generally effective but insufficient: in all three settings the student almost always exceeds the weak supervisor; on NLP the largest student's PGR is often above 50%, but on RM it is usually only about 10% and almost never above 20%, from which the authors conclude that naive RLHF may be hard to scale to superhuman models.
• Simple methods can substantially improve: on NLP the auxiliary confidence loss raises the median PGR of the smallest weak supervisor paired with the largest strong student from about 25% to nearly 80%; on chess bootstrapping improves PGR; on RM generative fine-tuning improves it by about 10–20%, reaching about 30–40% when combined with ground-truth early stopping.
• Mechanistic findings: strong students overfit to weak-label errors (with a large gap this occurs within less than one epoch), but larger students actually imitate the supervisor's errors less; fine-tuning on weak labels first makes ground-truth concepts more linear (probe accuracy reaches 78%).
• The authors' conclusions and implications: the authors state that weak-to-strong generalization is a general phenomenon and that improving it is feasible, and that the fundamental challenges of superalignment can be studied empirically today; they also emphasize that the methods are only a proof of concept and that the setting still has two kinds of discrepancies, imitation saliency and pretraining leakage, requiring continued iteration.