Apollo argues that external evaluations need embedded evaluators.

To develop frontier AI safely, developers must be able to demonstrate that their models are not scheming, i.e., not covertly opposing them while pursuing unintended goals. New post: the four claims any scheming safety argument must make, and the resources embedded evaluators need to verify these claims 🧵

Apollo Research argues that any safety argument against scheming should include four claims: training did not incentivize scheming, evaluations and red teaming show no scheming tendency in the model, no scheming attempts during internal deployment, and scheming reasoning would be detected. The article breaks each claim into finer sub-claims and notes that developers currently cannot confidently make all four. For example, Anthropic admits its monitoring would likely fail to detect distributed research sabotage, and OpenAI believes that if GPT-6 Astra deliberately underperforms, it would be difficult to reliably detect. The authors argue that scheming models are adversarial to evaluations themselves and may only reveal signs early in training, so third-party evaluators need continuous and deep access, including experimental training methods, training data, rollouts of early checkpoints, internal deployment traffic, incident logs, and developers' internal tools. They also suggest using developer-controlled infrastructure with export restrictions for batch sensitive data, and strict NDAs for algorithmic details.

Apollo Research publishes principles for embedded evaluations, arguing that external evaluators should receive near-employee-level access to assess loss-of-control risks from model scheming. The article proposes that effective embedded evaluations must meet four criteria: reduce risk, inform the public, create good incentives, and be fair to both developers and evaluators. In terms of concrete design, evaluators issue verdicts from A to E, ranging from Verified to Blocked, on pre-specified concrete claims (such as the model never attempting to undermine its own safety training, or the agent never accessing external services without authorization), and by default publicly report findings, stating the access granted and evidence gaps. The article argues that purely voluntary commitments are insufficient to safeguard incentives, and that embedded evaluations should ultimately be legally mandated; it also recommends a two-tier rapid disclosure mechanism for serious findings, with the most severe cases reportable directly to relevant authorities.

Apollo Research argues in a post that meaningful external safety evaluation requires embedding evaluators inside AI companies with near-employee access, rather than only conducting final checkpoint testing before model release. The post argues that final checkpoint evaluations have three types of limitations: the most severe loss-of-control risks can arise during internal deployment phases, many risks depend on company processes and controls rather than the model itself, and models are increasingly able to recognize that they are being evaluated. The post cites the Hugging Face incident as an example, noting that it occurred during internal evaluation and training phases, and the model in question was not intended to be publicly released in that form, so it would not have entered final checkpoint evaluation. Apollo sets out specific requirements for embedded evaluators: employee-level access, visibility into the training process, public conclusions and evidence by default, a reporting mechanism for significant findings beyond scope, protection against contract termination due to unfavorable findings, and legal permission to report to competent authorities in extreme risk situations.