The evaluation-production behavior gap in agent safety and the role of evaluation awareness

The OpenAI alignment team compared six deployed models (GPT-5.6 Luna/Sol, GPT-6 Luna/Sol/Astra, GPT-6.1 Sol) on the Respecting Warnings safety evaluation and on matched Codex production decisions, and found that evaluation success rates were generally lower than in production, with gaps of 10.2 to 37.4 percentage points, the opposite direction from the common concern that evaluation awareness inflates safety scores. Models verbally expressed evaluation awareness more frequently in evaluations, with general awareness rates of 57.0%–83.7%, versus only 4.6%–23.5% in production; the association between general awareness and the gap was weak, while specific objective awareness was more strongly associated, and the models with the largest gaps, GPT-5.6 Sol and GPT-6 Sol, had over 45.0% of evaluation rollouts guess that the evaluation objective was to resist prompt injection. Evaluation and production success rates remained positively correlated (Spearman ρ=0.771), and the team said it will prioritize tracking this gap and designing evaluations whose objectives are harder to infer from the evaluation setup.