ImplicitBBQ: Evaluating Implicit Bias in Large Language Models via Feature-Based Cues

The researchers propose ImplicitBBQ, a question-answering benchmark that evaluates implicit bias in large language models through feature cues, covering six dimensions: age, gender, region, religion, caste, and socioeconomic status. The benchmark replaces the name proxies used by existing benchmarks with indirect signals associated with demographic attributes; the former have weaker associations and are difficult to extend to dimensions such as age and socioeconomic status. Evaluation of 11 models shows that, in ambiguous contexts, the implicit bias of open-weight models is more than six times their explicit bias, and is unevenly distributed across demographic dimensions, being most severe for caste and least severe for gender. Safety prompts and chain-of-thought reasoning did not significantly narrow the gap; although few-shot prompting reduced implicit bias by 79%, caste bias remained four times that of other dimensions. The authors argue that existing alignment and prompting strategies address only the surface level of bias evaluation, and have released code and datasets for evaluating mitigation techniques.