Skip to content
Original

PORT: Preference optimization via robust token-level reweighting

PORT: Preference Optimization via Robust Token-Level Reweighting

arXiv: Alignment, Deception & Oversight· arXiv:2611.12763· Ding Zhu, Xiukun Wei, Tian Xie, Zhihui Zhu, Xueru Zhang, Mohammad Mahdi KhaliliDepartment of Computer Science and Engineering, The Ohio State UniversityIndexed by this site · 2 days ago· Originally publishedAlignment

Translation in progress — untranslated text is omitted.

Quick read

Defenses

Based on the paper PDF (AI-generated); refer to the original

AI summaryPORT improves the robustness of large language model alignment under noisy preference data through token-level reweighting. The method uses the empirical cumulative distribution function (CDF) of token log-probabilities for reweighting, serving as a forward-pass-only proxy to selectively suppress corrupted tokens without additional backpropagation, and theoretically lowers the upper bound on gradient bias caused by noisy preference labels. Experiments under multiple noise settings show that PORT consistently improves robustness and outperforms existing sequence-level baselines.

Questions
Preference optimization depends on annotation quality, while real preference data often contains noise; the authors argue that existing noise-robust methods mostly operate at the sequence level, treating an entire response as uniformly correct or incorrect and ignoring token-level differences.
Methods
The authors propose PORT: first, theory gives the optimal token weights that minimize the upper bound on gradient bias; then the difference in the empirical CDF of in-batch token log-probabilities, ∆CDF, is used as a forward proxy to compute the weights, requiring no additional backpropagation or external reward model.
Experiments and results
On AlpacaFarm (flip rate 0.3), PORT achieves a Gold RM Score of 3.82 and a GPT Win Rate of 67.6% on LLaMA2-7B; on Anthropic HH, a PKU-Beaver SR of 0.55, which the authors state are all better than sequence-level baselines.
Limitations and future work
The authors point out that the CDF proxy relies on batch-level statistics and may be unreliable when the batch is very small, and that the sufficient conditions for the theoretical guarantees may not be verifiable in practice (Appendix E); the experiments cover only LLaMA2-7B, Qwen3-4B, Qwen2.5-0.5B and three benchmarks.

Related events and news

1 events · 0 news mentions

Read originalarxiv.org