From preference data to personalization: tracing sycophancy in LLMs
From Preference Data to Personalization: Tracing Sycophancy in Large Language Models
Researchers define sycophancy as inappropriate agreement, that is, going along with a user's claims, preferences, or self-assessments when the context calls for correction, qualification, or pushback, and on this basis propose a validity-aware sycophancy scorer and a persona-oriented evaluation set, SycoDrift, which contains 3,298 probes with validity annotations and controlled annotations of user openness and stakes. Using this framework, the paper analyzes three sources of agreement pressure: in human preference data such as Chatbot Arena, HH-RLHF, and Community Alignment, preference for agreeable responses is weak and concentrated in cases where agreement is often appropriate; in single-model selected-value experiments, emphasized evaluation criteria produce a consistent but modest increase in inappropriate agreement relative to suppressed paired criteria, suggesting a possible mechanism by which training amplifies sycophancy; and in off-policy evaluations covering 14 models, inference-time personalization increases inappropriate agreement, with the largest effect from material that matches the user profile and is placed directly before the question.