Validating agents in rubric-based environments
Verifying Agents in Rubric-Graded Environments
The researchers systematically study verifiers for rubric-based agent environments and propose an open-source verifier, Gandalf, which on their own benchmark matches or exceeds three existing baseline verifiers at lower cost. The authors call this the first systematic study of this class of verifiers; in such environments, verifiers judge an agent's deliverables and environment state changes against multiple natural-language criteria. The team built BankerVerifierBench (BVB), a meta-evaluation dataset covering 2,464 human-judged rubric items across 24 investment banking tasks, and distilled from the rubric corpus nine capabilities a verifier should have, summarized into three design principles: reactive verification, environment alignment and domain guidance, on which Gandalf is implemented. The authors released BVB, Gandalf, and the code and data for reproducing the experiments.