Skip to content
LessWrong· Alek Westover·Indexed by this site · Originally published· Governance →· Its place in the scan record →

To limit the compute that AI companies use for capability advancement, researchers proposed a classification definition of capability activities

How should we define “capabilities activities” for the purpose of pacing?

AI summary

Researcher Alek Westover proposed a set of classification definitions for "capability activities," intended as a reference for restricting the compute AI companies use to advance AI capabilities through legislation or voluntary commitments. The definition counts AI company activities as capability activities by default, exempting only four categories: idle compute, narrow product development, serving customers, and narrow safety activities, all premised on not advancing general model capabilities. Narrow safety activities are limited to alignment evaluations, company safety testing, whitelisted safety research (interpretability, information content of chain of thought, model persona, model organisms that do not involve new algorithmic ideas, alignment theory with no near-term capability application), running monitors and abuse classifiers, and investigating serious misalignment incidents, among others; safety research must publicly disclose project plans and results reports. Scalable oversight, dangerous capability evaluations, Constitutional AI / RLAIF, training monitors or abuse classifiers, jailbreak robustness training, and improving RL environments are classified as capability activities. The author argues it is better to misclassify something as a capability activity than to miss one, and calls for third-party audits to measure and publicly report each company's compute allocation; the text mentions that Anthropic has published an estimate of the share of its safety research.

Read originallesswrong.com