Cohere proposes CulturalRiddles benchmark to evaluate cultural context reasoning with 6,200 riddles

Cohere researchers built the CulturalRiddles benchmark, collecting 6,200 original riddles from 62 cultural communities in 51 languages, and evaluated 28 model configurations. The task measures cultural context reasoning, and the authors note that it cannot be equated with capability on all real-world tasks. The overall best score of 79% masked regional gaps, with the best models for some communities not exceeding 22%. Among the answers the authors manually checked, about 31% were judged as confabulation; when allowed to answer "don't know," models used that option in only about 0.7% of responses, and the authors emphasize that this statistic applies only to the evaluation and annotation conditions in this paper. For five models on English translations of five difficult communities, the average score rose from 22% to 32%, and the authors believe language coverage is not the whole reason.