Skip to content
WeChat · AI Era (Xinzhiyuan)·Indexed by this site · Originally published· Evaluation →· Its place in the scan record →SelectedAttention44

Claude writes its own inference engine! Waiting for AI's first token, cut from 12 seconds to half a second

AI summary

Researchers from Carnegie Mellon University, NVIDIA and other institutions submitted the Singularity-Bench preprint on November 12, breaking AI R&D into three training tasks and three infrastructure tasks to test whether agents can independently optimize the AI R&D process. In the inference engine task, Claude Opus 5.5 with Claude Code, in an offline environment using 4 GB300s, wrote an inference engine; when replaying 16 concurrent real coding sessions, the P90 first-token latency was 0.478 seconds, lower than SGLang v0.5.20's 12.029 seconds, and the two had similar generation speeds. Among the 11 competing combinations, Opus 5.5 with Claude Code won five and tied one of the six tasks, becoming the only combination to exceed the human baseline (1 point) with a score of 1.252; second place, GPT-6.1 Sol with Codex, scored 0.969. The paper's authors acknowledge that truly testing recursive self-improvement still requires more time and compute.

Why it was selected

The benchmark breaks AI R&D into six closed-book questions and provides a human baseline, allowing readers to judge the current state of agents automating AI engineering and the distance to recursive self-improvement.

Related papers
Read originalmp.weixin.qq.com