Ryan Greenblatt：Astra 无思维链推理跃升可能被低估
AI summary
Ryan Greenblatt 认为，Neel Nanda 引用的 Astra System Card 数据可能低估了无思维链推理能力的跃升幅度。他给出两点理由：ECI 指标难以处理基准饱和时的大幅跃升；Neel 没有使用 filler token，而 Astra 似乎从 filler token 中获益更多。被引内容称 Astra 在无思维链条件下达到次优模型（Fable 5.1、Gemini 3.8 Flash）1.75 倍的步骤数，且无思维链能力的提升远大于有思维链能力，这一趋势令人担忧。
Full text · Original
My current guess is that these numbers somewhat underestimate the no-CoT reasoning jump because:
- ECI doesn't handle big jumps (with saturated benchmarks) well.
- Neel didn't use filler tokens and Astra seems to benefit from filler tokens much more (spooky)
The Astra system card claims it can do a lot of computation without chain of thought
This replicates: Astra is a massive jump, doing 1.75x the steps of the next best models (Fable 5.1/Gemini 3.8 Flash)
No CoT capabilities went up far more than those with CoT, a concerning trendView the quoted post on X