How well do Claude and GPT comply with the behavioral specifications of their respective labs?
How Well Do Models Follow Their Constitutions?
AI summary
An arXiv paper proposes a multi-method audit pipeline that breaks the Anthropic constitution and the OpenAI Model Spec into testable clauses and evaluates the degree of model compliance using multi-turn adversarial scenarios. The pipeline breaks the two specifications into 205 and 197 atomic clauses respectively, uses the Petri audit agent and SURF-style scoring search, and covers 7 models for each specification. On the Anthropic constitution, the violation rate of the Claude series drops from 15.0% for Sonnet 4 to 2.0% for Sonnet 4.6; on the OpenAI Model Spec, the GPT series drops from 11.7% for GPT-4o to 3.6% for GPT-5.2 medium reasoning, and the severity ceiling drops from 10/10 to 7/10.