SpecEval: Evaluating the degree to which models follow behavioral specifications

Researchers propose the SpecEval automated audit framework to test whether foundation models truly follow the behavioral specifications publicly disclosed by their vendors. Across 16 models from six vendors and more than 100 behavioral specifications, they found systematic inconsistencies, with compliance gaps between vendors reaching up to 20%. The framework parses specification statements, generates targeted prompts, and uses models to judge compliance; its core examination is the three-way consistency among vendor specifications, model outputs, and judges using the vendors' own models. The audited vendors include OpenAI, Anthropic, and Google, among others.