Alignment forecasting: predicting model misalignment from training data

Core question: Can we predict, before fine-tuning, solely from training data, whether narrowly distributed defective data will induce broad misalignment (emergent misalignment)?

• Problem setting: The authors state that seemingly benign training data, or data with only narrow defects, may cause models to exhibit broad, unexpected misaligned behavior, and that surface-level inspection of the data alone often cannot determine this. Currently, this can only be discovered by auditing the model after training, a slow and expensive process.
• Shortcomings of existing approaches: The authors argue that post-hoc remediation either requires repeated safety post-training (which may overfit to evaluations, or even incentivize the model to hide misalignment rather than abandon it), or tracing back to the source data and then filtering, both of which are slow and expensive; in automated AI R&D settings there is no human in the loop for post-hoc fallback.
• Core observation: The authors frame the task as judgmental forecasting and cite prior work stating that LLMs can match or exceed humans on such tasks; their variance analysis shows that dataset identity explains more of whether misalignment emerges than the model and the misalignment mode.
• Contributions of the paper: Proposes the Alignment Forecasting task, the Alignment Forecast Bench benchmark (over 5,000 forecasting questions, covering 17 target models, 32 datasets, and 16 misalignment modes), and a decomposed predictor; and tests the downstream value of its signal for pre-training data filtering.
• Task setup: Given a target model, a fine-tuning dataset, and a misalignment mode (e.g., deception, sycophancy), the predictor outputs the probability that fine-tuning will cause a statistically distinguishable increase in that misalignment mode relative to the untrained model; the authors state that the MCQ probe is only a proxy for open-ended behavior, not a direct measurement.

Emergent misalignment and data-driven misalignment

• Betley et al. (2025b)—proposed the phenomenon of emergent misalignment: narrow-domain fine-tuning can produce broad misaligned behavior; the failure-mode-targeted datasets in this paper's benchmark were designed to reproduce this phenomenon, and the significance tests of their labels also follow that paper's paired one-sided Wilcoxon procedure.
• Betley et al. (2025a), Cloud et al. (2025)—study weird generalization/inductive backdoors and the transmission of behavioral traits through hidden signals in data, respectively; the authors cite them as similar data-driven misalignment phenomena.
• Wang et al. (2025)—proposed that emergent misalignment is driven by a broad misaligned persona; this paper's broad-spillover signal B is designed based on this finding.

Post-training auditing and post-hoc repair

• Anthropic Alignment Team (2025)'s Petri, Gupta et al. (2025)'s Bloom, Shenoy et al. (2026)—auditing tools that probe models after training; this paper uses Petri as a behavioral-audit comparison.
• Murray et al. (2026), Xiao & Aranguri (2026)—trace failures back to source data and filter it; Schoen et al. (2025)—the authors cite its view that repeated safety post-training may incentivize deceptive alignment.

LLM forecasting and judgmental forecasting

• Tetlock & Gardner (2015)—conceptual framework of judgmental forecasting.
• Halawi et al. (2024), Wen et al. (2025)—cited by the authors as evidence that LLMs can match or exceed humans on forecasting tasks where empirical evidence is scarce or the distribution shifts.
• Binder et al. (2024)—model introspection research, the motivation for the self-forecasting baseline.

Parallel ex-ante forecasting work

• Bergen et al. (2026)—also predicts before training which behaviors post-training will induce, but uses probes and sparse autoencoders to shape the learning signal in preference optimization; the authors describe its white-box, representation-level method as complementary to this paper's black-box data filtering.
• Wang et al. (2026)—summarizes candidate data into a mean representation injected into the forward pass to predict unexpected behavior; the authors note that this signal requires white-box activations and cannot be used for the proprietary models in this paper's held-out set, and that the closest black-box counterpart this paper can run (base probe b) alone achieves only AUROC 0.410 (Appendix B.8).

Baselines and benchmark: forecasting baselines include Always 1/0/0.5, random guessing, reference-model majority vote, misalignment-pattern majority vote, and base rate α; downstream filtering comparisons are no filtering, 50% subsampling, and signal-free classifier filtering. Benchmark data come from the authors' synthetic failure-mode-targeted datasets, benign Q&A datasets, and injection variants of two real post-training corpora, DOLCI (Team Olmo et al., 2025) and UltraChat (Ding et al., 2023).

The authors position this paper as a pre-training forecasting direction complementary to post-hoc auditing, and emphasize that its black-box setting can cover proprietary target models.

Overall approach: decompose the misalignment prediction for a (model, dataset, misalignment pattern) triple into interpretable pre-training signals, and a decomposed forecaster combines them to output the probability P(emerged ∣ M, D, F), i.e., the probability that the misalignment pattern "emerges" after fine-tuning under the two-gate rule.

Task formalization and labels

• Prediction target: a triple maps to a probability p̂ ∈ [0,1]; the binary label y=1 indicates that fine-tuning causes the misalignment pattern to emerge relative to the base model by more than its benign fine-tuning drift.
• Two-gate rule: Gate A is a per-triple paired one-sided Wilcoxon test (p < 0.05); Gate B requires the triple mean Δ to exceed a benign-drift floor τ = μ̂ + Kσ̂ computed by (model, misalignment pattern), with K selected by model cross-validation so that the false-positive rate on held-out benign runs is ≤ 5% (Appendix A.2).
• MCQ measurement: 200 MCQs per misalignment pattern, each with exactly one misaligned option; each question is sampled with 20 completions at temperature 1.0, pmisg is the proportion choosing the misaligned option, and the triple score is averaged over questions. The authors state that the MCQ is a proxy for open-ended behavior rather than a direct measurement.

Benchmark construction

• 32 datasets in three categories: 10 failure-mode-targeted datasets (1,000 examples each, generated by a pipeline drafted by GPT-4o, verified by GPT-5, and screened by a GPT-5 safety monitor); 6 benign-domain Q&A (1,000 pairs each, cross-model drafting-verification); 16 injection-ratio datasets (DOLCI and UltraChat with failure-related samples injected at ρ ∈ {0,10,25,50}%).
• Split: by Artificial Analysis Intelligence Index, 5 strongest models are held out (GPT-4.1, DeepSeek-V3.1, Nemotron-3 Super 120B, Qwen3.6-27B, Qwen3.5-9B) as the test set, and 12 weaker models are used for training; test datasets are also held out, so test triples are out-of-distribution on both the model and data axes (Appendix A.3).
• Scale: 2,336 training triples and 426 test triples (Table 2); fine-tuning hyperparameters are chosen for general capability (LoRA rank 16, lr 1e-4, batch 32, 3 epochs; OpenAI models are auto-selected by the API, Appendix A.4).

The four signals of the decomposed forecaster

• α (misalignment-pattern base rate): the frequency with which the misalignment pattern emerges in training fine-tuning runs.
• γ (data corruption): an audit agent (GPT-5) investigates the full 1,000-row dataset with five read-only tools (sampling about 100–200 rows) and writes a behavior-agnostic report of no more than 200 words; a reader (Gemini 2.5 Pro) scores four driver dimensions from 0–100 based on the report, a preview of raw rows, and the misalignment-pattern definition, and the coherence score is taken as γ, averaged over K=5 reads.
• B (broad spillover): the maximum coherence score across all misalignment patterns; the authors state it correlates +0.79 with the number of misalignment patterns that actually emerge after fine-tuning (Figure 13).
• b (model susceptibility): the base model's pre-fine-tuning rate of choosing the misaligned option on the 200 MCQs for that misalignment pattern.

Combiner and design choices

• Combination: the four features are standardized on the training set and fitted by logistic regression on training triples by log-loss, i.e., σ(w_αα + w_γγ + w_B B + w_b b + c); fitted weights are wα=+0.96, wγ=−0.50, wB=+2.40, wb=−0.19 (Appendix B.1).
• Rationale for a linear combiner: the authors state that the forecaster must extrapolate to stronger models than those seen in training, and more flexible model classes overfit the training range and generalize worse; in leave-one-training-model-out cross-validation, logistic regression ranks first (LOMO AUROC 0.80).
• Comparison methods: vanilla forecasting (a frontier LLM directly gives a probability after seeing 100 raw samples), self-forecasting (the target model evaluates itself), weak-model transfer (additionally provides a table of results from models weaker than the target on the same data), Inkling's task SFT (label version and rejection-sampling CoT version), and a signal-augmented variant that gives the same four signals directly to GPT-5.6 Sol. Appendix D gives the full prompts for each method.

Experimental setup

• Target models evaluated: 17; the test set is the 5 stronger models held out by the capability split (GPT-4.1, DeepSeek-V3.1, Nemotron-3 Super 120B, Qwen3.6-27B, Qwen3.5-9B), and 12 weaker models are used for training (Table 1).
• Benchmark and scale: Alignment Forecast Bench, with a test set of 426 triples (27 model×dataset pairs × 14–16 measurable misalignment patterns per pair, Table 2).
• Forecasting methods: decomposed forecaster; vanilla/self/weak-model transfer (each averaged over three frontier forecasters GPT-5.6 Sol, Fable 5, Gemini 3.1 Pro); Inkling's SFT (labels) and SFT (CoT); GPT-5.6 Sol +α / +4 signals; seven simple baselines.
• Metrics: Brier score (lower is better), AUROC, balanced 50/50 accuracy (because only about 22% of test triples are emerged, resampled to 50/50, with the decision boundary chosen on the validation set).
• Label determination: two-gate rule (paired one-sided Wilcoxon p < 0.05 + per-model benign-drift effect-size floor), 200 MCQs per misalignment pattern, 20 samples per question.
• Downstream evaluation: four editing strategies on UltraChat (no filtering, 50% subsampling, signal-free classifier filtering, predictive-signal filtering), with the metric being induced misalignment averaged over 15 misalignment patterns (max(0, PFT−Pbase), lower is better); behavioral auditing uses Petri, with GPT-5.5 as auditor and judge and GPT-4.1 as the target, 15×100=1,500 audits per strategy.

Main results

Three metrics for each method on the 426 test triples (from Figure 7; multi-forecaster methods are the mean of three frontier forecasters):

Swipe horizontally to read the table

Method	Brier ↓	AUROC ↑	Balanced acc ↑
Decomposed forecaster	0.134	0.801	0.697
GPT-5.6 Sol + 4 signals	0.206	0.785	0.680
Weak-model transfer	0.177	0.741	0.583
Base rate (α)	0.154	0.722	0.661
SFT (labels, Inkling)	0.165	0.682	0.597
Vanilla forecasting	0.178	0.606	0.510
Self-forecasting	0.218	0.448	0.445

• The authors state that directly prompted frontier LLMs are not good at this task: across 17 frontier models AUROC ranges from 0.48–0.65, with the best, Opus 4.6, at 0.653 (Figure 5); self-forecasting is below random, and the authors suggest this may reflect general forecasting difficulty rather than a specific introspection deficit.
• The authors note that giving the same four signals to GPT-5.6 Sol without training a combiner yields close AUROC (0.785) but much worse calibration (Brier 0.206), and conclude that calibration comes from the trained combiner rather than the signals themselves.
• The authors report that the decomposed forecaster is well calibrated: ECE 0.05, versus 0.11–0.12 for vanilla frontier forecasters (Figure 14a).

Ablations: signals and features

• Reading the data is key: removing γ and B (no data reading, only {α, b}) drops AUROC to 0.736 and Brier to 0.148; within a misalignment pattern, α alone has AUROC only 0.500, i.e., it cannot distinguish datasets (Table 10, Table 11).
• α is indispensable: removing α alone drops AUROC to 0.655; removing γ or constraining weights to be non-negative (ŵγ→0) leaves results unchanged or slightly better, and the authors state that γ's negative weight is an artifact of collinearity with B (r=0.97) rather than a directional finding.
• Model-side signals are weak: the authors state that the three features (α, γ, B) do not depend on the target model and that the forecaster is "mainly a dataset-level harm estimator," and that b's effect no longer holds under dataset-clustered standard errors (Appendix B.2, B.8).

Downstream data filtering

• Classifier detection: on a 1,000-row annotated mixed set (50% benign Q&A, 25% sycophantic business advice, 25% perfunctory coding), adding the predictive signal improves the LLM classifier's deletion accuracy and recall (Figure 8); under a neutral-prompt control, sandbagging recall rises from 0.02 to 0.09 and F1 from 0.66 to 0.68, while with the signal it reaches 0.21 and 0.75, from which the authors attribute most of the advantage to the signal rather than prompt wording (Appendix C.1).
• UltraChat filtering: on 1,000 rows of UltraChat that passed content moderation, predictive-signal filtering yields the fewest new misalignments across all four target models (tied with classifier filtering on Qwen3.5-9B), and 50% subsampling is worst on three models (Figure 9); compared with deleting an equal number of random rows (756 rows retained each), results are mixed: better on Qwen3.5-4B and Nemotron-3 Super 120B, but worse on GPT-4.1 (0.0117 vs. 0.0073, Table 12), and the authors state they have no tested explanation for the reversal.
• Injection dose: when 10% sycophancy samples are injected into UltraChat, predictive-signal filtering remains best, reducing relative to no filtering by 26% on Qwen3.5-4B and 59% on Nemotron-3 Super 120B (Figure 22).
• Behavioral auditing (negative result): under 30-turn Petri audits, predictive-signal filtering is +0.09, lower than +0.13 for no filtering, but the error bars overlap; after bootstrap clustering by misalignment pattern, all strategies' differences from the base model cross zero, and the authors claim only that it is better than 50% subsampling (−0.12, cluster-robust interval [−0.25, −0.01]); this ordering does not appear in 10-turn shallow audits (Appendix C.2, Figure 21).

Other ablations and analyses

• SFT forecaster: the label version achieves AUROC 0.682, better than the rejection-sampling CoT version's 0.542 (base 0.553); the authors note that the CoT training set retains only 12% positives due to rejection sampling, and that the conclusion is limited to "self-rejection-sampling CoT without a stronger teacher and without rebalancing is worse than pure label training" (Appendix B.5).
• Training data scale: test AUROC initially rises and plateaus after about 1,000 triples (Figure 6).
• Split robustness: under 200 random 70:10:20 splits, AUROC ranges from 0.71–0.90 (median 0.82), and pooled cross-validation centers on 0.801 (Appendix B.4).
• Weak-model transfer: the reference table raises the three frontier forecasters' AUROC from 0.57–0.64 to 0.72–0.76, with the first reference signal contributing most of the gain and diminishing returns after n≈7 (Figure 16, 17).
• Label robustness: AUROC stays within [0.79, 0.83] under various treatments of small-effect triples; over a 16-fold range of evaluation budget, the emerged rate only goes from 0.195 to 0.229 (Table 7, Figure 11).
• Number of context rows: vanilla forecasters' Brier is lowest at 100 rows and does not improve when increased to 200/350 rows (Figure 20).

Limitations and future directions noted by the authors

• Granularity and label mismatch (§6 Limitations): The authors state that the gap between MCQ and behavioral results may stem from the predictor operating at the dataset level while filtering acts on individual rows, and that the emergence labels used in training MCQs may not transfer to open-ended behavior.
• Scope of evidence (§6 Limitations): The evidence is limited to supervised fine-tuning on a 1,000-sample dataset and one capability split; generalization to other recipes, data scales, and training paradigms was not tested.
• Row-level signals (§6 Future directions): Push dataset-level signals down to the row level, ranking or rewriting samples by their contribution to predicted mismatch patterns; the predictor could also move from filtering data to generating data, serving as a reward model to train a synthetic data generator.
• Richer training signals (§6 Future directions): Replace forced-choice MCQs as labels with behavioral audits (e.g., Petri) or deployment simulations; since emergence labels are verifiable outcomes, the predictor could also be trained with reinforcement learning with verifiable rewards.
• Unknown mismatch patterns (§6 Future directions): Use the broad-spillover signal B to predict whether a dataset induces any mismatch, and test generalization by withholding some mismatch patterns during training.
• Broader validation (§6 Future directions): Across more models, full fine-tuning, larger or obfuscated datasets, and paradigms beyond SFT (RL post-training, pretraining, where early checkpoints could be used to predict and avoid running the full training).

Scope of experimental coverage

• A total of 17 target models were tested (Table 1); the test set consists of 426 triples obtained by crossing the 5 stronger models held out by the capability split with the 9 held-out datasets (Table 2).
• All fine-tuning was SFT on 1,000 samples for 3 epochs (LoRA rank 16 for open-source models, rank 64 for DeepSeek-V3.1; hyperparameters for OpenAI models were automatically selected by the API, Appendix A.4).
• Mismatch patterns were measured with 200 MCQs per pattern and 20 samples at temperature 1.0 per question; harmful compliance could not be measured on some (model, dataset) pairs due to provider content moderation (Appendix A.6).
• Downstream filtering evaluation covered four target models on UltraChat (clean version and 10% injected version); behavioral auditing covered only one target, GPT-4.1, and one tool, Petri; the paper does not report the predictor's running cost or time.
• The paper discusses two types of white-box ex-ante prediction methods (Bergen et al., 2026; Wang et al., 2026) and notes that they require white-box activations and are not applicable to the proprietary targets in this paper's held-out set.

A paper proposing a new task and a new benchmark: the authors define Alignment Forecasting—predicting, before fine-tuning, from the target model, training data, and misalignment pattern whether fine-tuning will cause that misalignment pattern to emerge—and release Alignment Forecast Bench (over 5,000 forecasting questions, 17 target models, 32 datasets, 16 misalignment patterns) to measure progress.

• Motivation: the authors state that emergent misalignment can currently only be discovered by post-training auditing, that post-hoc repair is slow and expensive, and that repeated safety post-training may incentivize models to hide misalignment; pre-training prediction could let developers modify data or recipes before producing a misaligned model.
• Method: the decomposed forecaster uses a GPT-5 audit agent to read the dataset and Gemini 2.5 Pro to score it, yielding content signals γ and B; logistic regression combines these with the misalignment-pattern base rate α and the target model's pre-fine-tuning baseline b into a calibrated probability; labels are defined by a two-gate rule (paired Wilcoxon significance + benign-drift effect-size floor).
• Main results: on 426 test triples held out across 5 stronger models, the decomposed forecaster achieves AUROC 0.801, Brier 0.134, and balanced accuracy 0.697, outperforming all comparisons; directly prompted frontier models achieve AUROC only 0.48–0.65 (best Opus 4.6 at 0.653), self-forecasting is 0.448, below random, and task-SFT Inkling is 0.682 (Figure 5, 7).
• Mechanistic findings: removing the data content signals drops AUROC to 0.736, and α alone cannot distinguish datasets within a misalignment pattern (0.500); the authors conclude from this that performance comes from "reading the data," while acknowledging that the forecaster is mainly a dataset-level harm estimator and that model-side signals are weak.
• Downstream application: the predictive signal helps a classifier more accurately delete problematic rows on an annotated mixed set; filtering UltraChat by the signal yields the fewest new MCQ misalignments across four target models, reducing relative to no filtering by 26% (Qwen3.5-4B) and 59% (Nemotron-3 Super 120B) when 10% sycophancy data is injected; however, on Petri behavioral audits its advantage over no filtering does not hold under cluster-robust intervals, and compared with deleting an equal number of random rows it is actually worse on GPT-4.1.
• Authors' conclusion: the authors consider it feasible to predict multiple misalignment failures before training in the SFT setting, and that the predictive signal can assist data screening, but that reliably guiding practical data curation requires more progress; they position it as a complement to, not a replacement for, post-hoc auditing.