OpenAI 员工称 AI 杀死全人类概率超 10%，支持 AI 架构可监控性提案
AI 导读
X 用户 @balesni 自称在 OpenAI 工作，他表示自己认为 AI 有超过 10% 的概率杀死全人类，并称 Ryan Greenblatt 提出的关于 AI 架构可监控性与性能权衡的政策提案是行业降低该风险最应做的事之一，但认为这还不够。Greenblatt 称，他担心 AI 转向在不透明激活而非思维链中推理的“neuralese”架构；基于有限公开证据，Astra 似乎是朝这一方向迈出的令人担忧的一步，因此 AI 公司应公开相关证据和政策，并在世界其他地区有机会讨论证据和政策之前，对可能消除或大幅减少对思维链依赖的架构保持谨慎。
正文 · 原文
i am at OpenAI and i think AI is >10% likely to kill all humans
this proposal is among the top things we should do as an industry to lower that risk (it’s not enough though!)
I'm very worried about changes to AI architectures that result in AIs thinking in opaque activations instead of in chain of thought (aka "neuralese" architectures).
Based on limited public evidence, it seems like Astra was a concerning step in this direction.
Unfortunately, there was insufficient public information to have a well-informed public scientific discussion about whether the changes to architectures and training methods that went into Astra were a good trade-off between monitorability and performance. We also don't know how AI companies will make these trade-offs going forward. I think companies should release the evidence needed for a reasonably informed public conversation about how these trade-offs should be made. They should also make and publish their policies around this. We've written up a proposal for how this could work.
I don't know if this proposal will be sufficient to avoid the most concerning architectures, but it seems like a relatively robust step in the right direction. AI companies should be very cautious about pursuing architectures that could eliminate or greatly reduce dependence on chain of thought, and certainly shouldn't do this before the rest of the world has a chance to discuss the evidence and their policies.在 X 查看被引用的帖子