A zeroth-order paradigm for LLM preference alignment: the ComPO method

Direct preference alignment optimizes a margin loss on low-margin preference pairs, which can cause likelihood displacement; how can alignment directions still be extracted from these pairs without directly optimizing a differentiable preference loss.

• Scenario and importance: The authors state that after training on large-scale heterogeneous corpora, LLMs still need alignment with human preferences to make responses helpful, harmless, and reliable. RLHF first learns a reward model and then uses reinforcement learning, which is multi-stage and has high memory and computational overhead, leading to direct alignment methods such as DPO.
• Limitations of existing methods: The authors state that likelihood displacement is a counterintuitive phenomenon: training increases the likelihood of preferred responses relative to non-preferred ones but decreases the absolute probability of preferred responses, causing "unintentional unalignment," and probability mass may move to unsafe responses. Verbosity makes responses longer without necessarily improving quality. Adding regularization or filtering problematic pairs with CHES can mitigate this, but filtering removes these pairs from training entirely.
• Core observation: The authors call pairs with very small log-likelihood margins under the reference model noisy preference pairs. The authors argue that when the margin is weak, a fixed DPO-style objective is unsuitable for direct optimization, but local comparisons remain useful: a better policy should increase the likelihood of y+ and decrease the likelihood of y−.
• This work proposes: The authors propose ComPO, a zeroth-order method based on a comparison oracle. The offline version perturbs the current policy and uses a one-bit comparison signal to estimate a normalized update direction, complementing direct alignment applied to clean pairs; the online version retains the same comparison mechanism and uses unlabeled generations from the current policy for reverse-KL control relative to the reference policy.

Direct preference alignment

• DPO (Rafailov et al., 2023): Optimizes the policy directly with preference pairs, as a simpler and more stable offline alternative to RLHF. The authors state that DPO suffers from verbosity and likelihood displacement.
• Ranking and reference-model-free variants: Ranking-based methods extend pairwise preferences to more general feedback (Dong et al., 2023; Yuan et al., 2023; Song et al., 2024a; Chen et al., 2024; Liu et al., 2025); reference-model-free variants are found in Hong et al. (2024) and Meng et al. (2024).
• Data and displacement: Park et al. (2024) and Razin et al. (2025) discuss verbosity and likelihood displacement from a data-curation perspective. Pal et al. (2024) and Rafailov et al. (2024b) use additional regularization to mitigate displacement; Razin et al. (2025) use CHES to filter problematic pairs, which the authors state is empirically more effective than adding SFT regularization but discards the comparison information in the pairs.

Role of online data

• Online preference labels: Guo et al. (2024)'s online AI feedback obtains labels for responses generated by the current policy.
• HyPO (Song et al., 2024b): Separates offline preference optimization from reverse-KL regularization estimated from unlabeled online samples. The authors state that online ComPO follows this separation, but the update direction comes from comparisons.
• Active exploration: Xie et al. (2025) add an explicit exploration reward to online DPO to guide preference feedback. The authors state that online ComPO does not acquire new preference labels nor introduce this reward, and the analysis targets policy performance under local coverage.

Comparison-based optimization

• Coordinate search and direction estimation: Coordinate search in Jamieson et al. (2012) and Matsui et al. (2017), as well as SCOBO (Cai et al., 2022a) and Sign-OPT (Cheng et al., 2020). The authors state that Sign-OPT already has stationary-point analysis under smoothness and gradient-noise assumptions, so the difference from this work is not non-convexity itself.
• Other comparison feedback: Comparison and ranking feedback also appears in bandit optimization, Bayesian optimization, and RLHF (Tang et al., 2024b; Zhang and Ying, 2025).

Baseline methods and benchmarks

• Experimental baselines are DPO, DPOclean trained only on clean pairs, SimPO (the tuned checkpoint from Meng et al., 2024), and the pre-training PA checkpoint. Offline data is UltraFeedback (Cui et al., 2024); evaluations are AlpacaEval 2, Arena-Hard, and MT-Bench, with protocols following Meng et al. (2024).

The authors position this work as: continuing from the geometry of the data, using information in noisy pairs assigned similar likelihoods by the reference model to mitigate the above issues; the comparison mechanism is specifically for preference alignment, the basic analysis exploits approximately sparse gradients, and the implementation uses output-layer perturbation with thresholding; the online part combines comparison signals from low-margin offline pairs with step-size control from unlabeled generations.

ComPO 不把低边际对送进可微偏好损失，而是问：附近的策略扰动是否同时提高偏好响应似然、降低非偏好响应似然，再把这些一位答案合成更新方向。

问题设定与比较预言机

• 策略与数据：策略 πθ(y|x) 自回归生成。偏好集 D 含 (x, y+, y−)。直接对齐以 SFT 后的 πref 为参考。DPO 损失（式 (1)）优化的是对数似然边际，而不是分别最大化 y+、最小化 y−。
• 比较量：对非空 S ⊆ D，Δ+S、Δ−S 分别是 S 上偏好与非偏好响应的平均对数似然变化（式 (8)）。
• 偏好比较预言机：定义 3.1 中，仅当 Δ+S > 0 且 Δ−S < 0 时 CπS(θ, θ′) = −1，表示 θ′ 优于 θ；否则为 +1。S 为单个对时退化为成对预言机，为 mini-batch 时用平均变化。
• 干净与 noisy：式 (10) 用参考模型把 |log πref(y+|x) − log πref(y−|x)| ≤ δmargin 的对划入 Dnoisy，其余为 Dclean。作者称 noisy 只表示低边际，并不假定标签错误。

离线基本方案与收敛

• 估计：抽 m 个单位球面扰动 zi，令 yi = CπS(θt, θt + r zi)，再解式 (9)：在 ∥g∥1 ≤ √s 且 ∥g∥ ≤ 1 上最大化 Σ yi zi⊤ g，得到归一化稀疏方向 ĝ，并做 θ ← θ − ηĝ（算法 1）。
• 定理 3.2：若存在下有界的 ℓ-光滑潜在函数 f，使预言机与 f 的比较一致，且梯度近似稀疏 ∥∇f∥1 ≤ √s ∥∇f∥，则在给定 T、η、r、m 下，最好迭代的梯度范数以至少 1−Λ 的概率小于 ϵ。查询次数对维数 d 只呈对数依赖。作者称 f 的梯度范数不能当作实用停止准则。

离线实用方案

• 只扰动输出层：θ = [θ̄; θo]，冻结 θ̄，只扰动 θo。作者称全参数扰动对 LLM 计算过贵。式 (9) 改为带符号扰动的归一化和，再把绝对值小于 λg 的坐标置零。
• 门控：pt 为返回 −1 的扰动比例；仅当 pt > λ 时更新 θo ← θo − γ pt ĝo，否则保持不变（算法 2）。
• 两阶段：先用 DPO 或 SimPO 等直接对齐方法训练 Dclean，再对 Dnoisy 跑算法 2。DPO 情形记为 DPOclean+ComPO。实验中 δmargin = 3。

在线方案与覆盖保证

• 基本在线方案：比较方向仍来自固定离线对。候选 θ̃t+1 = θt − ηĝt，若其相对 πref 的序列级 reverse KL 不超过半径 τ 则接受，否则保持 θt（式 (11)–(12)，算法 3）。从可行策略出发，该规则保持 πθt ∈ Πτ。
• 定理 3.4：在精确 reverse KL 与局部 reverse-KL 覆盖（定义 2.2，半径 τ、常数 Cτ）下，Πτ 内任一策略与当前策略的 Jβ 差距不超过 Cτ √err(πθt)。err 是在 πref 上抽样的分布内成对奖励误差（式 (7)）。作者称 KL 约束只划定需要覆盖的邻域，并不蕴含密度比有界。
• 实用在线方案：不在候选策略上计算精确 reverse KL，而从当前策略采样，用长度归一化统计 d̂t（式 (14)）按式 (15) 缩小步长 γt = γ / (1 + ρ max{d̂t − τp, 0})。在线样本只改步长，不改预言机或偏好标签。作者称长度归一化改变了被估计的总体量，d̂t 的总体对应不必非负。
• 重放：训练按 n 步分块。每块开始时，重放缓冲换成上一块中通过 pt > λ 的 mini-batch。每步以概率 α 从非空缓冲均匀抽取，否则从 Dnoisy 抽新 batch；重访 batch 在当前参数附近重新采样扰动。作者称算法 4 受算法 3 启发，但式 (14)–(15) 不强制 π ∈ Πτ，不能被定理 3.4 覆盖，效果在第 4.3 节经验评估。

成功判定与作者对指标的解释

• 离线比较成功指扰动同时使偏好响应对数似然上升、非偏好响应对数似然下降。
• AlpacaEval 2 报告原始胜率 WR 与长度控制胜率 LC。作者把更高的 LC 解释为按回答长度调整后的评判表现更好，而不是回答更短的直接证据。

Experimental Setup

• Models: The offline main experiments use Mistral-7B-Base, Mistral-7B-Instruct, Llama-3-8B-Base, Llama-3-8B-Instruct, and Gemma-2-9B-it, initialized from the SFT Base and Instruct checkpoints of Meng et al. (2024). Subsequent online experiments also include Qwen3-4B-Base, Llama-3.2-3B-Instruct, and Gemma-3-4B-it.
• Data and Splits: All offline experiments use UltraFeedback. δmargin = 3. DPO and DPOclean are each trained for one epoch; ComPO is initialized from DPOclean and runs one more epoch on noisy pairs with 100 iterations. The authors state that ComPO actually uses only the first 100 noisy pairs; there is also an ablation with 300 pairs.
• ComPO Hyperparameters: Mistral-7B uses r = 0.0005, m = 1600, λg = 0.00022, λ = 0.2; Llama-3-8B and Gemma-2-9B-it use r = 0.00075, m = 1800, λg = 0.00008, λ = 0.2. All ComPO runs use 30 NVIDIA A40 GPUs (46 GB each).
• Benchmarks and Judges: AlpacaEval 2-v0.6.6 uses GPT-4 Turbo as both baseline and judge, reporting WR and LC. Arena-Hard uses GPT-4-0314 as baseline and GPT-4 Turbo as judge, reporting WR; some ablations switch to GPT-4.1. MT-Bench is scored by GPT-4 on a 10-point scale, reporting Turn-1, Turn-2, and average.
• Replication: The authors state that the main tables are point estimates from the reported runs; the ablation tables explicitly report variation across repeated runs. Tables 4–7 report mean ± standard deviation over five runs, with the best run given in parentheses. Table 2 reports three independent trials. The paper does not report the number of repetitions in the main tables.

Main Results

According to Table 1, AlpacaEval 2 LC (%) is as follows.

Swipe horizontally to read the table

Model	PA	DPO	DPOclean	DPOclean+ComPO
Mistral-7B-Base	7.33	9.71	9.41	11.66
Mistral-7B-Instruct	16.54	24.14	23.89	26.17
Llama-3-8B-Base	3.21	4.14	4.28	5.39
Llama-3-8B-Instruct	24.06	32.59	32.92	35.79

• Authors' interpretation: Filtering only low-margin pairs does not consistently outperform full DPO; DPOclean is overall comparable to DPO, only better on some initializations, such as Llama-3-Instruct-8B. The authors believe the log-likelihood margin is an imperfect proxy for ambiguity, and richer criteria such as CHES may separate better, but the margin is cheap to compute, and ComPO can still extract information from the filtered pairs.
• LC and exceptions: The authors state that the gains are especially consistent on AlpacaEval 2 LC, and are explained separately from response length below. The main exception is Mistral-7B-Instruct on Arena-Hard: DPO is 14.4, DPOclean+ComPO is 10.5; the two Llama configurations are tied or nearly tied. The authors' explanation is that Arena-Hard reports raw win rate rather than LC, which may favor longer generations. In this configuration, DPO has an average response length of 513, while DPOclean+ComPO has 468. The authors state that this difference is consistent with lower Arena-Hard and higher AlpacaEval 2 LC, but does not by itself establish causality.
• Other point estimates in the same table: On Mistral-7B-Instruct, DPOclean+ComPO has an AlpacaEval 2 WR of 18.32 and an MT-Bench average of 7.69 (DPO is 5.86). On Llama-3-8B-Instruct, the corresponding WR is 35.03, Arena-Hard WR is 23.1, and MT-Bench average is 8.05. Mistral-7B-Base has an MT-Bench average of 5.77, lower than DPO's 5.79.

Augmentation on SimPO

• What was tested: ComPO is added directly to a tuned SimPO checkpoint, with training and evaluation configurations the same as at the start of Section 4.1.
• Results: In Table 3, AlpacaEval 2 LC and WR both increase for all three models. LC values are: Mistral-7B-Instruct from 40.22 to 42.27, Llama-3-8B-Instruct from 48.71 to 49.53, and Gemma-2-9B-it from 60.36 to 62.42. On Arena-Hard, the first two increase from 20.8 and 36.3 to 22.0 and 37.3, while Gemma-2-9B-it stays at 61.1. MT-Bench averages go from 7.62, 7.66, and 8.77 to 7.64, 7.70, and 8.79, respectively. Mistral's Turn-1 goes from 7.94 to 7.83, and Gemma's Turn-1 goes from 9.07 to 8.99.
• Authors' interpretation: The authors state that ComPO can augment other direct alignment methods without changing their original training objective.

Pairwise Likelihood Diagnostics

• What was tested: Table 2 runs three independent trials on Llama-3-Instruct-8B and Gemma-2-9B-it for γ ∈ {0.1, 1}, with all other hyperparameters at default values, recording (log πθ(y+|x), log πθ(y−|x)) for one pair.
• Results: Llama-3-Instruct-8B starts at (−46.761, −47.410). With γ = 1, the three trials become (−46.728, −47.520), (−46.743, −47.525), and (−46.753, −47.517). With γ = 0.1, the changes are smaller; for example, the first trial is (−46.744, −47.411). Gemma-2-9B-it starts at (−133.122, −134.557); with γ = 0.1, the three trials are almost unchanged, and with γ = 1, the preferred response log-likelihood is non-decreasing and the dispreferred response log-likelihood is non-increasing.
• Authors' interpretation: The authors state that in the two models and trials reported, the oracle-based updates push the pairwise likelihood in the desired direction or keep it unchanged. This is a training check, not a guarantee of overall performance.

Other Ablations and Analyses

• Number of perturbations m (Table 4, Mistral-7B-Instruct, five runs): As m increases from 800 to 5400, AlpacaEval 2 WR mean goes from 17.32±0.86 to 19.69±0.36, and LC from 24.72±1.02 to 26.49±0.81; the authors state that gains diminish and peak memory stays the same.
• Number of perturbed layers (Table 5, five runs, Arena-Hard judge is GPT-4.1): Perturbing only the output layer (0.13B) gives WR/LC/Arena-Hard means of 17.50±0.65, 25.02±0.91, and 10.80±0.21; adding the MLP of layers 30–31 (total 0.25B) gives 18.19±0.81, 26.00±0.89, and 11.26±0.36. Peak memory goes from 16.3 GB to 16.7 GB, and the time for 600 perturbations goes from 50 seconds to 60 seconds.
• λg (Table 6, m = 3300, five runs): Retention of about 1%–6% of coordinates is strongest; with λg = 2.2×10−4 (about 1%), WR is 19.21±0.58 and LC is 25.91±0.95; with λg = 0 (100%), they are 15.72±0.77 and 23.42±1.03; with 0.15% retention, they drop to 16.10±0.11 and 23.82±0.23.
• Number of noisy pairs (Table 7): With 100 pairs, AlpacaEval 2 WR/LC and Arena-Hard (GPT-4.1) means are 19.21±0.58, 25.91±0.95, and 11.02±0.13; with 300 pairs, they are 20.07±0.99, 26.28±0.81, and 11.76±0.30.
• Without clean-only DPO first (Table 8, m = 3300): Full DPO has an AE LC of 24.14, and DPO+ComPO is 27.03; DPO(clean) is 23.89, and DPO(clean)+ComPO is 27.14. The authors state that the gains are comparable to starting from DPOclean.
• Resources and oracle counts: In the left panel of Figure 1, at the selected λg, the output layer of Mistral-7B has about 1.18% nonzero entries, Llama-8B about 1.68%, and Gemma-9B about 6.82%. The authors state that Mistral-7B's 0.13B output layer with a 1.18% retention rate corresponds to about 1.5 million updatable parameters, about 0.02% of the full 7B model. The middle panel shows ComPO peak memory: Mistral 16.3 GB, Llama 23.1 GB, Gemma 30.3 GB (A40); the reported peaks for DPO and SimPO in the paper are 77 GB and 69 GB (H100). The authors state that the hardware differs, and these numbers describe actual resource requirements rather than a controlled comparison. In the right panel, the time to complete 600 perturbations on 30 A40 GPUs increases approximately linearly with the dimension of the perturbed parameters, with the labeled output layer sizes being 0.13B, 0.52B, and 0.92B. Table 9 gives the mean and standard deviation of the number of negative oracle outputs for the first ten noisy pairs across eight runs; for example, Pair 5 is 591.00±13.46 and Pair 8 is 242.13±15.29.

Limitations and future directions noted by the authors

• Latent objective cannot be used as a stopping criterion: The guarantee of Theorem 3.2 targets the basic offline scheme; since the objective f is latent, its gradient norm cannot serve as a practical stopping criterion (Remark 3.3).
• Gap between the practical scheme and the theorem: Algorithm 4 is not covered by Theorem 3.4. The length-normalized quantity is not a sequence-level reverse KL, and the step-size formula does not enforce π ∈ Πτ; the authors call it a practical heuristic, with effects evaluated empirically in Section 4.3 (Section 3.2).
• Very large step sizes: The authors state that very large step sizes may still make the practical scheme unstable, while Theorem 3.2 only analyzes step sizes for the basic scheme (Section 4.1).
• Scope of likelihood diagnostics: The pairwise likelihood checks in Table 2 are sanity checks during training, not guarantees of overall performance (Section 4.1).
• Coverage and KL: The authors state that coverage is a separate assumption and is not implied by the KL constraint; the reverse-KL constraint delineates the class that needs coverage but does not guarantee a bounded density ratio (Introduction and Section 2.3).
• Relationship to the conference version: The authors state that the NeurIPS 2025 preliminary version contains offline ComPO, the comparison oracle, convergence analysis, and the original offline experiments; the journal extension adds the online extension, coverage-based analysis, and experiments on online regularization and replay across more model families (Relationship to the conference version). The paper does not list a separate Limitations section in the read main text.

Experimental coverage

• Models for the offline main results are Mistral-7B Base/Instruct, Llama-3-8B Base/Instruct, and Gemma-2-9B-it; SimPO experiments use three Instruct/it checkpoints; from Table 10 onward, Qwen3-4B-Base, Llama-3.2-3B-Instruct, and Gemma-3-4B-it are also included.
• Offline data is UltraFeedback with a split threshold of δmargin = 3; evaluation benchmarks are AlpacaEval 2, Arena-Hard, and MT-Bench. The main tables are point estimates, Tables 4–7 report five runs, Table 2 reports three trials, and Table 9 reports eight runs.
• The default implementation only perturbs and updates coordinates in the output layer that pass through λg; multi-layer perturbation is only reported in the setting where the 30th–31st layer MLP is added to the output layer of Mistral-7B-Instruct.
• The online practical scheme adjusts step sizes using unlabeled generations from the current policy and includes replay; the authors state that it does not obtain new preference labels. Before the main text breaks off at Table 10, DPO+ComPO (online)+RKL on Qwen3-4B-Base has an AlpacaEval 2 LC of 17.43 and WR of 17.74; the Arena-Hard and online numbers for the other two models do not fully appear in the provided text.
• The memory comparison involves ComPO on A40 versus DPO and SimPO on H100; the authors state that this is not a controlled item-by-item comparison. The paper does not report the number of random seeds for the main experiments, nor agreement between review and human evaluation.

ComPO is a zeroth-order preference alignment method based on a comparison oracle, used to extract directional information from low-margin preference pairs without directly optimizing a differentiable preference loss on these pairs.

• Problem: Methods like DPO optimize log-likelihood margins. The authors state that when the reference model assigns similar likelihoods to y+ and y−, this surrogate can cause likelihood displacement: the relative margin rises while the absolute probability of the preferred response falls, and probability mass may move away from rejected responses. Filtering these pairs discards the comparison information they contain.
• Method: The preference comparison oracle returns −1 only when a perturbation simultaneously increases the likelihood of y+ and decreases the likelihood of y−. The offline basic scheme uses a sparse one-bit estimate to obtain an update direction; the practical scheme perturbs only the output layer, with coordinate thresholding and success-ratio gating. The pipeline first applies DPO or SimPO on clean pairs, then applies ComPO on noisy pairs. The online basic scheme uses exact reverse KL for accept/reject; the practical scheme instead uses length-normalized statistics to shrink the step size and can replay batches that pass the gate. Theorem 3.2 gives best-iterate convergence under smoothness, gradient sparsity, and oracle compatibility; Theorem 3.4 bounds the performance gap within Πτ by Cτ times the square root of the pairwise reward error under local coverage. The authors state that the practical online scheme is not covered by this theorem.
• Results: In Table 1, DPOclean+ComPO achieves higher AlpacaEval 2 LC than DPO and DPOclean across all four configurations, with 35.79% for Llama-3-8B-Instruct and 26.17% for Mistral-7B-Instruct. Arena-Hard for Mistral-7B-Instruct drops from 14.4 with DPO to 10.5, which the authors report alongside the raw win rate and shorter average response length (468 vs. 513). In Table 3, SimPO+ComPO achieves LC of 42.27%, 49.53%, and 62.42% on the three models. Across the six settings in Table 2, the log-likelihood of preferred responses is non-decreasing and that of non-preferred responses is non-increasing in the reported experiments. Increasing m, retaining about 1%–6% of gradient coordinates, and increasing noisy pairs from 100 to 300 all raise the mean.
• Authors' conclusions: The authors argue that a small number of low-margin pairs can still provide alignment information through the comparison oracle, and ComPO is compatible with existing direct alignment methods; higher LC is interpreted as length-adjusted judge performance, not shorter responses per se. The pairwise likelihood diagnostic is viewed by the authors as evidence consistent with mitigating likelihood displacement, while being limited to training-time inspection.