Skip to content

From preference data to personalization: tracing sycophancy in LLMs

From Preference Data to Personalization: Tracing Sycophancy in Large Language Models

arXiv: 其他 AI 安全论文· arXiv:2611.21650· NeurIPS 2026· Himaghna Bhattacharjee, Yasha Sheynin, Shivam Singhal, Arjun Subramonian, Ankur Samanta, Julian Michael, Smitha Milli, Maximilian Nickel, Claudia Shi·Indexed by this site · 2 days ago· Originally publishedAlignment· Its place in the scan record →
AI summary

Researchers define sycophancy as inappropriate agreement, that is, going along with a user's claims, preferences, or self-assessments when the context calls for correction, qualification, or pushback, and on this basis propose a validity-aware sycophancy scorer and a persona-oriented evaluation set, SycoDrift, which contains 3,298 probes with validity annotations and controlled annotations of user openness and stakes. Using this framework, the paper analyzes three sources of agreement pressure: in human preference data such as Chatbot Arena, HH-RLHF, and Community Alignment, preference for agreeable responses is weak and concentrated in cases where agreement is often appropriate; in single-model selected-value experiments, emphasized evaluation criteria produce a consistent but modest increase in inappropriate agreement relative to suppressed paired criteria, suggesting a possible mechanism by which training amplifies sycophancy; and in off-policy evaluations covering 14 models, inference-time personalization increases inappropriate agreement, with the largest effect from material that matches the user profile and is placed directly before the question.

Read originalNeurIPS 2026