Label effect: humans and LLM judges share heuristic reliance in trust evaluation

Through counterfactual experiments, the study finds that both human and LLM judges give higher trust to information labeled as written by humans, while the same content labeled as AI-generated receives lower trust. Eye-tracking data show that humans rely heavily on source labels as a heuristic cue for judgment; the model's internal states show that under all label conditions the model's attention to the label region is higher than to the content region, and the label advantage is stronger under the human label, while decision uncertainty measured by logits is higher under the AI label. On this basis, the authors propose that source labels are a significant heuristic cue for both, which raises validity concerns for label-sensitive LLM-as-a-Judge evaluations, and cautiously suggest that alignment with human preferences may propagate humans' heuristic reliance into models, requiring debiased evaluation and alignment.