DocOps: A verifiable benchmark for autonomous agents in complex document manipulation

The researchers propose DocOps, a benchmark delivered as containerized Harbor bundles, for measuring the completeness and reliability of agents in manipulating documents in native formats such as spreadsheets, presentations, and PDFs, classified along two axes: operation (content, format, structure) and difficulty (L1 atomic to L4 cross-document state tracking). The benchmark replaces LLM-as-a-judge with deterministic verifiers that directly inspect output files, using structural predicates, linguistic anchors, and preservation predicates, achieving 95.31% agreement with human review; in 180 controlled mutation stress tests against 36 verified outputs, it detected 96.67% of violations.