Skip to content

Translation in progress Some entries still include text in the original language.

Redwood Research · News

Oct 13 · Tuesday

Redwood Research · News

Latest publications and developments
Redwood Research

Research topics

Prompt injectionAgent safetyDeception and schemingReward hackingChain-of-thought monitoringInterpretabilityAI controlBackdoors and poisoning隐私与数据泄露Supply-chain securityDangerous capabilitiesAI 与网络攻防开源模型安全Red teaming

Companies and institutions

OpenAIAnthropicGoogle DeepMindGoogle ResearchGoogle 安全MetaxAIMicrosoftNVIDIADeepSeek阿里巴巴 · Qwen智谱 · GLM月之暗面 · Kimi字节跳动腾讯蚂蚁集团上海人工智能实验室UK AISIUS CAISIMETRApollo ResearchRedwood ResearchFAR.AIPalisade Research

Reports and frameworks

System CardFrontier safety frameworks
TodayOct 13Tue
  1. 量子位, The Decoder · 19 sources量子位 · 19 sources · 27 reports · ↑368

    The turmoil of departures and firings in OpenAI's safety team

    In October 2026, David Robinson, who was responsible for OpenAI's safety transparency function and led the drafting of the Preparedness Framework, left the company and wrote in The Atlantic criticizing that the safety culture at the company and in the industry had collapsed, arguing that frontier labs should have multiple layers of redundancy like nuclear power plants. Around the same time, safety researchers Jasmine Wang, Tomek Korbak, and Mikita Balesni were fired. The three wrote to the board and the safety committee, denying the company's allegations that they mishandled sensitive information, saying the real reason was raising safety concerns, and calling for preserving the ability to monitor AI chain-of-thought. OpenAI responded that an internal investigation found the three violated sensitive information handling policies, constituting a significant breach of trust, and that the firings were unrelated to raising safety concerns, and said it was finalizing a contract with a third-party safety evaluation organization, with details to be announced in the coming weeks. Subsequently, Alex Turner, Amber Scorah, and Karl Koch wrote an opinion article for Fortune arguing that OpenAI cannot self-regulate because it fired employees who raised safety concerns.

    Latest developmentsTotal 9 developments

    1. Opinion article criticizes OpenAI for firing safety researchers and opposing self-regulation

    2. Neel Nanda questions OpenAI's explanation for firing three safety researchers

    3. OpenAI responds to firing of three safety researchers

Oct 11Sun
  1. CNBC · TechnologyCNBC · Technology · ↑157

    Third-party AI safety evaluation organizations step into the spotlight

    According to CNBC, Anthropic and OpenAI have committed to embedding third-party evaluators internally, bringing small nonprofit evaluation organizations such as METR and Apollo Research to the center of AI safety governance. Anthropic CEO Dario Amodei last month proposed providing evaluators with desk space, badge access, company laptops, and access roughly comparable to that of the internal risk assessment team, and allowing them to publish critical findings. OpenAI subsequently released its own plan. Funding and independence remain unresolved issues; METR said it has received about $71 million in commitments over the past six months.

Oct 1Thu
  1. Apollo Research, X @ApolloResearchApollo Research · 2 sources · 4 reports · 47

    Apollo argues that external evaluations need embedded evaluators.

    Apollo Research has published a series of posts arguing that meaningful external safety evaluations should embed evaluators inside AI companies with near-employee access, rather than only doing final checkpoint evaluations before release. The posts argue that final checkpoint evaluations have three types of limitations: the most severe loss-of-control risks may arise during internal deployment, many risks depend on company processes and controls rather than the model itself, and models are increasingly able to recognize that they are being evaluated. They cite the Hugging Face incident as an example, noting that it occurred during internal evaluation and training, and the model in question was not intended to be publicly released in that form. Apollo then proposes principles for embedded evaluation, requiring evaluators to obtain near-employee-level access and to meet four criteria: reducing risk, informing the public, creating good incentives, and being fair to both developers and evaluators. Evaluators issue verdicts from A to E for pre-specified concrete claims, ranging from Verified to Blocked, and by default report publicly, stating the access obtained and the evidence. Apollo also argues that any safety argument about scheming should include four claims: training did not incentivize scheming, evaluations and red-teaming show the model has no tendency to scheme, no scheming was attempted during internal deployment, and scheming reasoning would be detected. It notes that developers currently cannot confidently make all four claims.

    Latest developments

    1. Apollo proposes an embedded evaluation framework for scheming tendencies

    2. Apollo Research proposes core principles for embedded evaluation

    3. Apollo argues that external evaluations need embedded evaluators.

Sep 28Mon
  1. Buck Shlegeris · 39

    Ryan Greenblatt 加入 METR，继续开展类似 Hugging Face 报告的调查

    Ryan Greenblatt 宣布加入 METR，继续开展类似其 Hugging Face 报告的调查。他表示，当前大量与灾难性风险高度相关的 AI 开发基础信息并未公开，而近期事件让他改变了对公开信息价值的怀疑态度，认为获取 AI 公司内部经核实的信息尤为紧迫。他提到，现有有限的公开证据与一种可能性相符，即临近的递归自我改进可能大幅加速能力进展，进而可能在 6 个月到一年内产生极端超人类通用能力，并带来相应的大规模最坏结果风险；更多经核实的公开信息可帮助判断这类极端结果在近期是否更可能或更不可能。除能力与起飞外，对齐、安全、控制以及 AI 公司内部风险相关流程的公开证据同样有限。METR 初期计划聚焦能力/起飞、对齐与控制，他希望其他团队覆盖安全、内部流程等领域。Buck Shlegeris 表示与 Ryan 共事约 5 年，认为他此举是正确的，这些调查有望揭示失准风险。

  2. Ryan Greenblatt · 25

    Ryan Greenblatt 加入 METR 调查 AI 风险

    Ryan Greenblatt 宣布加入 METR，继续开展类似其 Hugging Face 报告那样的调查工作。他认为当前关于 AI 开发的大量基础信息未公开，而近期事件让他改变了对公开信息价值的怀疑态度；获取 AI 公司内部经核实的信息尤为紧迫，因为有限公开证据与“即将到来的递归自我改进可能大幅加速能力进展、甚至在一半年内产生极端超人类通用能力”的可能性相符。METR 初期将聚焦能力/起飞、对齐与控制，他希望其他团队覆盖安全、内部流程等领域。

Dec 9, 2024Mon
  1. FAR.AI · 13

    FAR.AI Bay Area Alignment Workshop 2024: 160 researchers focus on AI safety and alignment

    On October 24-25, 2024, 160 researchers and leaders from academia, industry, government, and nonprofit organizations gathered in Santa Cruz for the FAR.AI Bay Area Alignment Workshop, discussing evaluation, robustness, interpretability, and governance. Google DeepMind's Anca Dragan opened with "optimization misalignment," METR's Beth Barnes emphasized rigorous evaluation and open-source benchmarks, and Redwood's Buck Shlegeris introduced AI control strategies such as trusted monitoring and collusion detection. The workshop also covered topics including China's AI policy, sparse autoencoders and causal abstraction, and debate-based scalable oversight.

You have reached the end