OpenAI analyzes signal and noise in coding evaluations

OpenAI released a new analysis pointing out reliability and accuracy issues in the popular coding benchmark SWE-Bench Pro, raising questions about AI model evaluation results. The analysis's theme is separating signal from noise in coding evaluations.