Stanford CRFM calls for a coordinated third-party flaw disclosure mechanism for general-purpose AI
General-Purpose AI Needs Coordinated Flaw Reporting
Stanford CRFM publishes the paper "In House Evaluation Is Not Enough," calling on AI developers to invest in the flaw investigation needs of independent third-party researchers and to establish new standards for researcher protection, reporting, and coordination infrastructure. The paper, co-authored by 34 researchers from machine learning, law, security, social science, and policy, offers four contributions: a standardized AI Flaw Report for researchers, legal provisions developers can adopt to protect good-faith research, a flaw bounty mechanism developers can implement, and a recommendation for a centralized body to coordinate and triage flaws across stakeholders. The article notes that independent AI evaluation lacks legal protection and that there is no mechanism for responsible disclosure after finding flaws, leading many flaws to go unreported, companies unable to fix them, or only one AI developer being informed while other affected companies are ignored.