Skip to content
Mother Jones·Indexed by this site · Originally published· Governance →· Its place in the scan record →

OpenAI's "independent evaluation" does not look independent at all

OpenAI's "independent assessment" doesn't look independent at all

AI summary

OpenAI announced on Monday that it had invited four research organizations—METR, Apollo Research, Transluce and Goodfire—to evaluate its internal models and test the effectiveness of safety features, and Mother Jones wrote that this arrangement can hardly be called independent. The article listed the terms of each organization's announcement: OpenAI can approve evaluation team members and designate which employees they can access; Apollo Research's four-month residency evaluation allows OpenAI to withhold information on grounds of technical infeasibility, safety risk or intellectual property protection; Transluce can only access pre-approved models, safety documents and researchers; and METR's evaluation scope is also narrower than envisioned in its October alignment evaluation proposal. The article argued that this falls short of the "employee-level access" Altman promised in September, and reviewed the investigation after the July Hugging Face safety incident: researchers were in total only permitted six days of on-site work, and the report acknowledged having to delegate a large amount of the analysis to AI agents whose judgment and reliability are clearly inferior to human researchers.

Read originalmotherjones.com