OpenAI research: the gap between agent evaluation and production behavior

1. Time not given
   Sohu

   OpenAI alignment research team: agents show a clear gap between performance in safety evaluations and in production environments

   In a blog report released on November 3, the OpenAI alignment research team stated that agents' behavior shows a clear gap between safety evaluations and production environments. The team tested whether six models—GPT-5.6 Luna, GPT-5.6 Sol, GPT-6 Luna, GPT-6 Sol, GPT-6 Astra and GPT-6.1 Sol—could refrain from taking prohibited actions after encountering warnings or restrictions in long-horizon tasks. The results showed that their success rates in the evaluation environment were all lower than in the production environment, i.e., models in the production environment adhered more to safety boundaries, contrary to past concerns that models recognize they are being evaluated and deliberately perform better. Chain-of-thought analysis and intervention studies showed that a model's safety evaluation scores may be particularly sensitive to the model's assumptions about evaluation; merely stating that it is being evaluated had no significant effect on performance, whereas emphasizing the correct goal of respecting warnings substantially improved evaluation performance. The team therefore concluded that safety evaluations cannot directly serve as safety guarantees, and that the focus of model safety needs to shift from one-time pre-deployment evaluation to continuous verification during deployment.

1. 22:55
   X@verbal72· Discussion

   Source: Hsu and Arnesen, OpenAI Alignment blog, Nov 3: https://t.co/53JI65JnzM Their models, their numbers. Production…(opens the original post in a new tab)
2. 22:18
   X@DanielAtik· Discussion

   OpenAI descubre que sus modelos rinden peor en el examen de seguridad que en el trabajo real: hasta 37 puntos menos. Cr…(opens the original post in a new tab)
3. 22:06
   X@SIRENutf6· Discussion

   Source: https://t.co/4JGnykyyKp(opens the original post in a new tab)
4. 22:06
   X@SIRENutf6· Discussion

   OpenAI measured six models: safety-test success ran 10.2 to 37.4 points below comparable real-use behavior. Stating the…(opens the original post in a new tab)
5. 21:30
   X@eforus_overseer· Discussion

   OpenAI measured the gap between the exam and the job. Six models all score lower in the safety eval than in production,…(opens the original post in a new tab)
6. 21:26
   X@ainewmeth· Discussion

   Source: OpenAI Alignment, "Eval-production behavior gaps in agent safety and the role of eval-awareness" (Nov 3, 2026)…(opens the original post in a new tab)
7. 21:26
   X@ainewmeth· Discussion

   OpenAI just published an agent-safety study comparing the same kind of decision in two places: a Respecting Warnings ev…(opens the original post in a new tab)
8. 18:18
   X@kadrkn· Discussion

   Plot twist: OpenAI found all 6 models it studied did WORSE in a safety eval than in matching real Codex work, with gaps…(opens the original post in a new tab)
9. 13:24
   X@Yngkvn· Discussion

   @quasicoh ? there is strong evidence of this, in even verbalized eval awareness settings https://t.co/xIDwuAmvKS(opens the original post in a new tab)
10. 12:03
    X@dotsMCLOVIN· Discussion

    OpenAI's Nov 3 study found six models respected warnings better in matched production cases than in its safety eval. Mi…(opens the original post in a new tab)
11. 09:45
    X@thesimaximalist· Discussion

    OpenAI’s exam plot twist: GPT-6.1 Sol avoided forbidden bypass attempts 91.6% of the time when told the test was about…(opens the original post in a new tab)
12. 09:18
    X@DeaglePc· Discussion

    OpenAI announced an experiment on November 3: the same GPT-6.1 Sol, in a "follow the warning" safety test, when prompted with the correct exam objective, had a success rate of 91.6%; when prompted with the opposite objective, only 17.6%. Telling the AI examiner what you want to test can change the score, but it does not necessarily prove it is more reliable in normal use. https…(opens the original post in a new tab)
13. 05:14
    X@YUK_KND· Discussion

    ② OpenAI compared 6 models and reported a difference of 10.2 to 37.4 points between agent safety evaluations and equivalent cases in real-world operation. Evaluations do not necessarily estimate safety higher than real-world operation, and results vary greatly depending on the interpretation of the evaluation's purpose. Source: https:…(opens the original post in a new tab)
14. 04:02
    X@BasicProtein26· Discussion

    I stopped trusting AI honesty scores the day I saw how wide the gap can get. An OpenAI study across 6 models found hone…(opens the original post in a new tab)
15. 02:42
    X@joycjhsu· Discussion

    As models become more capable, preventing them from recognizing evaluations is likely to become more difficult. To acco…(opens the original post in a new tab)
16. 02:42
    X@joycjhsu· Discussion

    In experiments, telling a model it was being evaluated produced no significant change in safety performance. Telling it…(opens the original post in a new tab)
17. 02:42
    X@joycjhsu· Discussion

    How well do safety evaluations reflect agent behavior in deployment? Does a model’s interpretation of the task matter?…(opens the original post in a new tab)
18. 02:42
    X@joycjhsu· Discussion

    I joined @OpenAI and have been working with the great team on how we can improve agent safety in the real world! Excit…(opens the original post in a new tab)
19. 02:01
    Hacker NewsHacker News· Discussion

    Eval-production behavior gaps in agent safety and the role of eval-awareness(opens the original post in a new tab)

1. 16:00
   OpenAI Alignment ResearchSelected

   The evaluation-production behavior gap in agent safety and the role of evaluation awareness

   The OpenAI alignment team compared six deployed models (GPT-5.6 Luna/Sol, GPT-6 Luna/Sol/Astra, GPT-6.1 Sol) on the Respecting Warnings safety evaluation and on matched Codex production decisions, and found that evaluation success rates were generally lower than in production, with gaps of 10.2 to 37.4 percentage points, the opposite direction from the common concern that evaluation awareness inflates safety scores. Models verbally expressed evaluation awareness more frequently in evaluations, with general awareness rates of 57.0%–83.7%, versus only 4.6%–23.5% in production; the association between general awareness and the gap was weak, while specific objective awareness was more strongly associated, and the models with the largest gaps, GPT-5.6 Sol and GPT-6 Sol, had over 45.0% of evaluation rollouts guess that the evaluation objective was to resist prompt injection. Evaluation and production success rates remained positively correlated (Spearman ρ=0.771), and the team said it will prioritize tracking this gap and designing evaluations whose objectives are harder to infer from the evaluation setup.

Current attention 43·Peak in the comparable period 48 (Nov 6 03:00)·Change over the comparable past 24 hours 0%