Tampered ML: Evaluating LLMs' Judgment of Machine Learning Research

Researchers introduce the Tampered ML benchmark, which uses 1,858 pairs of experiment proposals to measure LLMs' ability to identify flaws in machine learning methods; frontier models score above chance but remain far from reliable. The research motivation is automating part of alignment research, which requires systems to perform reliably on hard-to-supervise tasks such as identifying flaws in proposals and judging whether an experiment can support its conclusions. Each pair in the benchmark consists of an experiment proposal reconstructed from a published machine learning paper and a counterfactual version that makes only minimal changes to introduce a specific methodological flaw; the model must choose the methodologically more rigorous proposal under two presentation orders. Human validation estimates that 86% of items are valid (95% CI: 81-91%). Opus 5 accuracy is 77.9%; models are highly sensitive to presentation order and prefer flawed designs that appear more specific, transparent, or conventional.