BEAKER: An expert-annotated benchmark for embodied brains in self-driving chemistry laboratories

Researchers propose the BEAKER benchmark to evaluate multimodal large language models as embodied brains in self-driving chemistry laboratories, containing 1000 expert-annotated question-answer samples covering 11 types of image and video tasks. The samples were manually created and reviewed by chemistry and embodied AI experts, without AI generation or annotation, and the tasks cover affordance localization, trajectory planning, equipment and state recognition, safety recognition, apparatus setup, operation and step reasoning, and temporal process reasoning. The study evaluated 28 general, scientific, chemistry-specific, and embodied MLLMs in an offline single-turn setting, and the results show that these models can follow coarse experimental procedures in videos, but still struggle to ground actions onto actionable regions and feasible trajectories, to recognize fine-grained laboratory equipment states and safety norms, and to order steps for medium-length procedures.