Redwood Research proposes capability activity definition standards for limiting frontier AI progress by compute
How should we define “capabilities activities” for the purpose of pacing?
Redwood Research published a set of capability activity definitions for use in proposals to slow frontier AI progress by limiting compute deployment, with implementation paths possibly including legislation or voluntary commitments. The definition by default counts all AI company activities as capability activities, with only four exceptions: idle compute, narrow product development, serving customers, and narrow safety activities, provided they do not improve general model capabilities. The narrow safety activity whitelist covers alignment evaluations, company safety testing, five categories of safety research (interpretability, CoT informativeness, model persona, model organisms, alignment theory), runtime monitoring and abuse classifiers, and incident investigation, where safety research must publicly disclose project plans and results reports; dual-use safety work such as scalable oversight, reward models, dangerous capability evaluations, constitutional AI, and jailbreak robustness training still counts as capability activities. The authors argue it is better to misclassify as capability activity than to miss it, and call for third-party auditors to measure and publish each company's compute allocation; the text mentions that Anthropic has published an estimate of its safety research share, but the definition is opaque and unaudited.
The original source proposes an adjudicable classification standard for capability activities for compute rate-limiting proposals, and discusses the boundary-drawing between narrow safety activities and dual-use safety work.
- Verbalizable representations form a global workspace in language models
- Gradient Routing: Localizing Computation in Neural Networks via Masked Gradients
- Model Spec midtraining: improving how alignment training generalizes
- Emergent Misalignment: Narrow Task Fine-Tuning Can Make Large Language Models Broadly Misaligned