MMDM: A Benchmark for Measuring Large-Scale Multitask Decision Making

The researchers propose the MMDM (Massive Multitask Decision Making) benchmark, containing 5,448 questions across 16 task categories, for evaluating System-1 decision models that directly output structured probabilistic decisions in a single forward pass without autoregressively generating reasoning traces. The questions combine newly constructed tasks with reorganized rewrites based on original materials, covering scenarios such as agent state tracking, tool and skill selection, preference judgment, and rubric scoring, with option spaces ranging from 2 to 1,024 and inputs up to 64,817 tokens.