Faithful or Confabulated? A Causal Framework for Rationalization Bias in LLM Review

The researchers propose a causal framework to test the cue invariance of LLM review, finding pronounced cue-anchored rationalization in reviews under label and placebo perturbations. Their proposed PROOF-BEFORE-PREFERENCE (lock evidence first, score, then rank) mitigation method markedly improves cue invariance over the baseline. The framework designs five cue interventions—Blind, Truth, Flip, Placebo, and Reveal-After—as well as metrics quantifying outcome anchoring and rationale anchoring, including label-aligned rhetoric and explanation drift, supplemented by consistency and stereotype-intrusion checks. The experiments use a new dataset of 1000 summaries generated by traditional extractive models and LLMs; the authors also design anchoring attacks using verbosity and confidence cues and compare mitigation effects with structured chain-of-thought prompting.