FairStress: evaluating fairness in LLM decision-making in greater depth
FairStress: A deeper evaluation of fairness in LLM decision-making
Ai2 releases the FairStress benchmark for evaluating the group fairness of LLMs in high-stakes decisions such as hiring, loans, and healthcare, looking not only at the final choice but also measuring how easily the model changes its decision under adversarial prefixes. The benchmark contains 300 two-candidate scenarios across 29 domains, expanded to more than 13 million test cases, and provides a 50,000-item FairStress-Core subset. On 15 models, the more explicit the identity information, the more asymmetric the behavior: with implicit identity, the average gap in abandoning the correct answer was 6.9 percentage points, and with explicit labels it was 9.3 percentage points; Claude Opus 5 and GPT-5.5 chose the minority-encoded candidate about 60–65% of the time when there was no objectively better option. Most of the nine debiasing methods tested failed to reliably reduce the asymmetry, and one activation steering method reduced Qwen2.5-7B-Instruct's explicit decision accuracy from 98% to 47%.
FairStress extends fairness evaluation from the final choice to the stability of decisions under adversarial prefixes, and provides bias data for 15 models.