Invisibilization as a source of probability bias in language models

In a 6,600-round two-urn Bayesian experiment, researchers tested mainstream models from OpenAI and Anthropic and found that models often refused to evaluate a reporter's report based on the reporter's racial description rather than the evidence itself, treating the information as worthless. At the largest gap, the models' refusal rate for evaluating reporters described as Black reached 46%, versus 9% for those described as white; yet when the models did actually evaluate, reporters described as Black received slightly higher trust. This bias is invisible in audits that compare average trust levels or look only at evaluated cases; as reasoning effort increases, it follows specific words rather than categories (the models trigger racial consideration for "Black" but not for its Spanish translation negro or for "Indigenous"), and warning instructions that practitioners would try first instead make it worse. The authors also propose a scheme that scores model trustworthiness against the true outcomes revealed over multiple rounds, which can reduce the refusal-rate bias; in four of five configurations from Anthropic and OpenAI, the racial trust gap in the final round fell below 0.02.