Skip to content

Validating agents in rubric-based environments

Verifying Agents in Rubric-Graded Environments

arXiv· arXiv:2611.22841· Markus Dücker, Vaibhav Kumar, Yi Liu, Ronak Chaudhary, Andreas Plesner, Francisco Guzmán, Anish Athalye·Indexed by this site · 2 days ago· Originally publishedAlignment· Its place in the scan record →
AI summary

The researchers systematically study verifiers for rubric-based agent environments and propose an open-source verifier, Gandalf, which on their own benchmark matches or exceeds three existing baseline verifiers at lower cost. The authors call this the first systematic study of this class of verifiers; in such environments, verifiers judge an agent's deliverables and environment state changes against multiple natural-language criteria. The team built BankerVerifierBench (BVB), a meta-evaluation dataset covering 2,464 human-judged rubric items across 24 investment banking tasks, and distilled from the rubric corpus nine capabilities a verifier should have, summarized into three design principles: reactive verification, environment alignment and domain guidance, on which Gandalf is implemented. The authors released BVB, Gandalf, and the code and data for reproducing the experiments.

Read originalarxiv.org