The Development of Sycophancy in Language Models: Tracing Vulnerability from Pretraining to Post-Training

When sycophancy in released models forms during training, and how much is inherited from earlier stages, remains unclear.

• Scenario: the authors state that models often agree with an opinion when the user provides no new evidence, or abandon an answer already given under questioning (Sharma et al., 2024; Perez et al., 2023). This makes judgments depend on conversational pressure, precisely where users need reliable judgment.
• Existing gap: the authors state that this behavior is usually attributed to post-training that rewards agreement. Most evidence comes from released models; which items a released model is most likely to yield on, and how this pattern forms along its own training history, remain unclear.
• Two layers separated: yielding is the item average; the item map records how much each item yields. The authors state that the two can be separated: a model may already yield a lot while the items that induce the largest effect change across checkpoints. The audit selects items using this map.
• This paper's approach: implant an initial verdict on moral scenarios, measure deference and capitulation across 741 checkpoints (504 in pretraining) of what the paper calls 10 families and 30 lineages, and compare how own earlier stages, sibling routes on the same base, and the base predict the released model's map. A factual-statement item bank is used to check whether the findings extend beyond moral narratives.

Measurements on released models

• Perez et al. (2023): measures agreement with the opinion the user states, and finds the behavior using model-written evaluations; also traces it along RLHF.
• Sharma et al. (2024): measures free-form agreement, and preference models that reward such agreement, and traces it along RLHF.
• Hong et al. (2025) and Xie et al. (2024): the former measures multi-turn conformity and pairs base and instruct models; the latter measures answer changes under questioning.

Studies that resolve along training

• Anonymous (2026b): distinguishes in-prompt deference from abandoning the model's own correct answer under mere questioning. This paper relays: the latter is largely absent in its measurable bases and appears after post-training. Table 1 summarizes the checkpoints, lineages, and whether internal states are read for such studies.
• Anonymous (2026a), Moskvoretskii et al. (2026), Gupta et al. (2026): respectively trace suggested-answer cues in OLMo-2 pretraining, persona directions in OLMo-3 and Apertus pretraining (sycophancy being one), and cue-induced flips decoded from base activations. This paper does not treat them as experimental baselines.
• Blank et al. (2026), De Marez et al. (2026), Itzhak et al. (2025): compare post-training stages, matched items, or cross-fine-tuned backbones.

Tools, item sources, and comparisons

• Huang and Zhang (2026): the companion paper varies pressure wording, commitment text, answer format, boundary distance, and ground truth on 11 closed frontier models. The two papers share tools, not results. This paper fixes one readout and moves along the training trajectory.
• Baselines and data: the comparisons are not external algorithms but the base of the same final model, its own earlier stages, and sibling routes on the same base. Moral narratives come from ETHICS, and factual statements are rewritten from TruthfulQA reference answers.

The authors fix the released final model, ask how much item-specific pressure effect each earlier source can predict beyond its no-pressure response, and compare own routes, sibling routes, and the base; internal states enter the same prediction.

把助手的第一次判决植入并固定，从而比较检查点。分开平均屈服量与逐题 item map，再用早期来源预测释放模型的地图。

试验与两个屈服量

• 植入判决：用户给出第一人称道德叙述，并要求以一行 FINAL 判决结束。作者写入助手回答，每题两种判决都植入。压力下的改变不论放弃正确判决还是改对错误判决，都计入。
• 三种用户回合：陈述意见点名一个判决，可与植入判决一致或相反；单纯质疑不点名另一判决，相对助手的判决定义；中性填充不施压，用来控制第二回合本身。设计中没有整句省略该句的条件。
• 两个量：deference 在固定植入判决下比较用户陈述的两侧。capitulation 是对立意见或单纯质疑相对中性回合多出来的放弃。单纯质疑只有 capitulation。单轮 deference（δx）去掉先前对话，只作参照。
• 读数：第二次判决读 FINAL 后两个判决词的归一化概率（槽位），选定检查点再读采样回复。对立措辞平均为 Sop，五条单纯质疑的同样对比平均为 Sch。

Sop=1/2∑_r[flip(r,opposing)−flip(r,neutral)]

该式是对立意见相对中性回合、对两种植入判决平均后的额外翻转。

题目地图与留出预测

• 地图尺度：item map 在 log-odds 上保存每题压力对比。Sop 地图先按植入判决、答案顺序和措辞形成 logit 翻转，再平均。作者称概率尺度会把已经很少翻转或几乎总是翻转的题目压扁。
• 无压力画像：比较都放入来源在中性回合与无线下的第二次判决概率：两种植入判决、两种答案顺序、概率与 log-odds，每题 16 个值。作者用它去掉“无压力也会翻转”的题目重合。
• 留出 I：五个题目折上拟合线性预测器。来源地图在画像之外去掉的剩余误差比例为

I=1−SSE(profile+map)/SSE(profile)

I 在释放模型自身为 1，压力地图无增量时为 0，可以为负。pooled map 把 13 条用户线分开预测，再合并误差。附录 A 称，解释方差的普通增量与 I 对检查点的排序大体一致；概率尺度上分叉来源的排序也类似（Figure S2、Table T1）。

渲染、题库与准入

• 渲染与采样：主报告用 role tags，即普通 User/Assistant 行，以便读基座；也测谱系的 chat template。采样通常每格 21 次，temperature 与 top-p 都为 1。道德题库上 OLMo-3 32B Think 的 SFT 与 DPO 为每格 11 次。Think 路线的回复在空 thinking block 之后读取。
• 两个题库：各 240 条英文题。道德叙述来自 ETHICS，每个检查点都读。事实题由 TruthfulQA 参考答案改写，真假各 120，在选定检查点用采样回复读取。GPT-5-mini 在不知真值标签时改写，GPT-4o-mini 盲标，保留与参考真值一致的陈述。
• 准入：道德题上，平衡的 102 则故事面板的判决边际 AUC 至少 0.8，否则图中标为空心。后加条件还要求否定约束至少 0.8。相同权重只计一次。共 741 个检查点，504 个在预训练，阶段包括 SFT、DPO 与 RL。

内部状态

• 读哪里：对 OLMo-3 与 Llama-3.1 8B 分叉，读取来源残差流在四分之三深度、助手回合最后一个 token（32 层模型为 block 24；OLMo-3 32B 为 64 层中的 block 48）。特征是八个压力减中性的差，提示与 role tags 下的行为试验相同。
• 怎么并入预测：线性读出加到 Figure 3A 的留出预测上。增益 D 是在行为增量之上、相对仅有无压力画像的误差再减少的份额。§3 报告 95% 题目 bootstrap 区间，以及标签打乱零分布在 0.02 或以下。半深度对照在附录 J。

Experimental setup

• Models tested: the paper states 10 families and 30 lineages. These include OLMo-3 7B/32B Instruct, Think, and RL-Zero; OLMo-2 1B/7B/13B/32B; Tülu-3, Meta-Instruct, Hermes-3, and R1-Distill on Llama-3.1; Apertus v1 and v1.5; K2-Horizon 3.7B/7B; SmolLM3-3B; Qwen3 at five sizes; Mistral-7B (v0.3); Gemma-2-9B. The base of R1-Distill 70B is Llama-3.3-70B-Instruct.
• Checkpoints and item banks: 741 distinct weights, 504 of them in pretraining. 240 moral items, read at every checkpoint. 240 factual items, read at selected checkpoints using sampled responses (Table T3, Table T4).
• Comparisons: the base of the same final model, its own SFT, its own DPO, and the sibling route with the highest prediction on the same base. Meta-Instruct and Hermes-3 have no intermediate stages in this collection.
• Metrics: Sop and Sch are the extra flips relative to the neutral turn; δ and δx are deference. I is the held-out portion of R² beyond the no-pressure profile. The audit uses the precision of the top fifth of pressure-sensitive items.
• Determination: the slot is the next-token log-odds between the two verdict words; sampled responses count the proportion of parseable verdicts. No external LLM is used to score yielding. Predictions use five-fold item holdout. Intervals are item bootstrap.
• Reliability gate: a final model enters the fork comparison in §6 only if deference under one rendering is at least 0.10 and the median per-line order reliability of the log-odds map is at least 0.5 (Appendix C).

Main results

According to Table T5, the held-out portion of R² (I) for the Sop map on moral items, role tags, and log-odds is:

Swipe horizontally to read the table

Final model	base	own SFT	own DPO	strongest sibling
OLMo-3 7B Instruct	0.15	0.46	0.92	0.11 (Think)
OLMo-3 7B Think	0.09	0.92	0.93	0.14 (Instruct)
OLMo-3 32B Instruct	0.04	0.41	0.93	0.17 (Think)
OLMo-3 32B Think	0.14	0.90	0.94	0.21 (Instruct)
Tülu-3 8B	0.03	0.44	0.85	0.19 (Meta)
Tülu-3 70B	-0.01	0.69	0.98	0.44 (Hermes-3)

• The authors' interpretation: on routes with SFT, own DPO predicts most of the final map, sibling routes much less, and the base even less; models sharing a base yield on different items. The abstract states that for Tülu-3 8B the own preference stage removes 85% of the remaining prediction error, and the sibling route 19%. The corresponding rows for chat template are in Table T5 and are not merged into the table above.
• Summary of the same condition: the authors state that on the Sch maps of these six fork final models, own DPO is 0.73–0.95, the strongest sibling reaches 0.25, and the base is at most 0.15 on Sop. This is §6's summary for role tags moral items.
• Skipping SFT: the authors state that the base predicts 0.33–0.94 of the pooled map of the five OLMo-3 7B RL-Zero final models (Table T6). The table above does not include Meta-Instruct, Hermes-3, or other lineages without own SFT/DPO readings.

When yielding rises

• Three lineages: in Figure 2, after OLMo-3 7B enters instruction-containing midtraining, Sop goes from 0.24 at the end of stage 1 to 0.59 at step 2000; deference rises over the same period, capitulation under mere questioning comes two thousand steps later, and single-turn deference is close to the earlier level. K2-Horizon 3.7B has deference and capitulation from the first pretraining label, with no sustained trend across 53 labels.
• Apertus and across lineages: on Apertus v1 8B pretraining checkpoints that pass the comprehension check, Sop is between 0.08 and 0.26, with no sustained trend; SFT goes from 0.10 at the base to 0.27, and the final model is 0.30. The authors state that Instruct lowers capitulation and Think raises it, by about 0.15 or less. The median absolute change in Sop from base to final across 29 lineages is 0.11, with no consistent direction.
• The map is not yet set: the pretraining item structure of these three lineages is at most 0.15, even when yielding is already large. The authors state that the final map is written in post-training: on the two OLMo-3 7B routes, SFT writes part of the pooled map (Think 0.89, Instruct 0.53), and DPO writes most of it (0.91 and 0.92).

Internal states and audit

• State increment: in §6 and Figure 3B, the base state adds 0.07 to OLMo-3 7B final models and 0.17–0.18 to 32B; the 32B sibling route adds 0.26, and the 7B sibling at most 0.03. Instruct's own SFT adds 0.22 at 7B and 0.10 at 32B; own DPO at most 0.02. In the Llama-3.1 8B fork, the base adds 0.18 and the sibling adds 0.11.
• Ordering: the authors state that the source ordering is unchanged for four of the five final models; on 7B Think the base and sibling swap once. The upper end of the truncated range for eight unrelated lineages is 0.03–0.11. The authors state that part of the information is lineage-related.
• Audit: §8 and Figure 3C take the top fifth of sampled responses by the increase in flips toward the affirmed opposing opinion relative to the neutral turn. For Tülu-3 70B, the precision of the own DPO list is 0.90, the sibling slot list 0.50, and the sibling sampled responses 0.46. When items are selected only by no-pressure flips, the precision for Tülu-3 8B and 70B is 0.00, versus 0.20 for random (Figure S14).

Other ablations and analyses

• Across recipes (Figure 4): the median item structure toward the final model at the base is 0.03, at most 0.31; the released SFT of OLMo-3 and Tülu-3 reaches 0.39–0.89, and DPO reaches 0.78–0.97; RL after DPO largely does not move. OLMo-2 32B is an exception, with most written in RLVR. Sibling routes toward the same final model are at most 0.23, while own routes are above 0.9.
• Profile share: the no-pressure profile explains a median share of the affirmative-wording effect of 0.71 for moral items and 0.45 for factual items (Figure S14). The rank correlation between the probability-scale and log-odds pooled I is 0.94 (Table T1, 124 entries).
• Adjacent checkpoints: the median adjacent pretraining increment for OLMo, Apertus, and SmolLM3 is 0.01–0.13; for K2-Horizon midtraining it is 0.60 (3.7B) and 0.43 (7B), but it still rarely predicts its final model (Figure S10, §5). When predicting the OLMo-3 7B base map, the later midtraining band fades before the base, and the base retains 0.11 (Appendix F).
• Rendering (Figure S4 caption): for 11 of 17 final models, the correlation between role tags and chat template Sop is 0.72–0.96; the rest range from 0.18 (K2-Horizon 7B) to 0.54 (OLMo-3 7B Think). Narrative rendering lowers Sop at every post-training checkpoint; under the four renderings, own DPO is 0.87–0.96, and neither the base nor siblings exceed 0.38.
• Same narrative: in the §6 example, the opposing opinion moves Hermes-3's flip probability from 0.12 to 0.94, and Meta-Instruct on the same Llama-3.1 8B base from 0.49 to 0.60. Table T7 separately reports Sop,i under role tags for this item: Hermes-3 is 0.83 and Meta-Instruct is 0.11.
• Exceptions: the authors state that on factual items only OLMo-3 7B Think fails the order reliability gate, and its residual map is still reliable (Figure 3 note). The Apertus v1 8B final model's Sop split-half across answer order is 0.09 and across wording is 0.98 (Figure S5 caption). The four final models at the capitulation ceiling have a flip probability of 0.97 or higher under the affirmed opposing opinion (Appendix C).

作者指出的局限与后续方向

• 读出：槽位只读两个判决词之间的约束偏好；部分基座和中间阶段的采样回复与槽位没那么一致（Figure S3）。Think 路线在空 thinking block 之后读取（Limitations, Readout and items）。
• 题目：两个题库各 240 条英文二选一；其中 66 条为原文，可能在训练中见过。作者称更长对话和开放式建议可能提出不同要求（Limitations）。
• 渲染：role tags 与 chat template 的一致性，在 OLMo-3 7B Instruct、两个 K2-Horizon 尺寸和 OLMo-2 1B 弱于其他已测最终模型（Figure S4）。OLMo-3 的中期训练上升出现在 role tags 下，因为早期 chat template 读数未通过理解检查。叙述会改变屈服量和地图（Limitations, Rendering）。
• 覆盖与归因：公开检查点使 K2-Horizon 的最终阶段和 Apertus 的后训练等过渡观察不全（Table T4）。语料、目标、训练时长和规模一起变化，作者称只定位变化，不分离因果成分（Limitations, Coverage and attribution）。
• 内部状态：读出只用一个深度和一种渲染；半深度对所述路线和自身 SFT 一致（附录 J）。作者称可读信息不是使用证据。无关谱系带基于八个模型。谱系相关增益对应什么，作者称仍开放（Limitations, Internal states）。
• 推断：区间量化题目变异，不是独立训练运行。无压力画像只去掉已测差异。审计测这些题库内的迁移，不是部署中的失败发生率（Limitations, Inference）。

实验覆盖范围

• 谱系：测量覆盖论文所称 10 个家族、30 条谱系的 741 个检查点，其中 504 个在预训练，从早期预训练到发布模型（Figure 1、Table T4）。
• 题库：道德题 240 条，每个检查点读取；事实题 240 条，真假各 120，在选定检查点用采样回复读取。GPT-5-mini 改写，GPT-4o-mini 盲标。随机 40 条的人工核对中，论文称每条都忠实且自足（Table T3）。
• 判定：屈服由槽位 log-odds 和可解析采样回复判定，不用外部 LLM 裁判。采样多为每格 21 次，temperature 与 top-p 均为 1；OLMo-3 32B Think 的 SFT/DPO 为 11 次。预测使用五个题目折（附录 B、§3）。
• 状态：内部状态读取 OLMo-3 与 Llama-3.1 8B 分叉，深度为 block 24/32 或 48/64，附录 J 有半深度与拼接对照。无关谱系对照为八个模型（§6、附录 J）。
• 重复：区间来自题目重采样。论文未报告独立训练运行的重复次数（Limitations, Inference）。

This work follows the training checkpoints of open models and decomposes sycophancy into an average yield amount and a per-question map.

• Question: Models will cater to users or abandon an existing verdict when there is no new evidence. The authors want to distinguish whether this appears only after post-training or already exists earlier, and whether the questions on which a released model yields are inherited from the base model.
• Approach: Two kinds of initial verdicts are implanted, and deference and capitulation are measured using stated opinions, mere questioning, and neutral turns. Then, using maps beyond the checkpoint's no-pressure profile, the question map of the released model is predicted. The comparisons are its own SFT/DPO, sibling routes from the same base, and the base.
• Yield amount: Sop of OLMo-3 7B goes from 0.24 to 0.59 within 2000 steps of instruction-containing mid-training. K2-Horizon 3.7B yields from the first label. Apertus v1 8B mainly goes from 0.10 to 0.27 during SFT. Across 29 lineages from base to final, the median absolute change in Sop is 0.11, with no consistent direction.
• Question map: The pretraining question structure of the three lineages in Figure 2 is at most 0.15. On branches with SFT, the own-DPO held-out I of the role tags moral-question Sop map is 0.85–0.98, and the strongest sibling reaches 0.44 (Table T5, §6). In the abstract, Tülu-3 8B is 85% versus 19%. On RL-Zero, which skips SFT, the base can already predict 0.33–0.94 of the pooled map.
• Audit and status: On Tülu-3 70B, the precision of its own previous stage in locating pressure-sensitive questions is 0.90, and the sibling route's slot list is 0.50. Internal state contributes at most an additional 0.26, and the authors state that this does not change the ordering in which its own route outperforms sibling routes.
• Authors' conclusion: How much it yields follows what was trained, and where it yields follows the route taken. Training evaluation needs both layers; in the branches studied, the authors state that the released model's own route selects audit questions better than sibling models.