Beyond g-mean²: OpenAI proposes supplementary metrics for measuring model monitorability
Beyond g-mean²: complementary metrics for monitorability
OpenAI Alignment Research released two monitorability metrics supplementary to g-mean²: first, a pairwise matching metric computed only on instances supported by both models and aggregated into a model ranking; second, a metric for deployment risk when the monitor acts as a rejection gate. The pairwise matching metric uses inverse-variance-weighted HodgeRank to fit pairwise comparisons into a ranking, reducing task-subset selection bias caused by differences in model capability; in synthetic experiments it recovered a preset monitorability order of seven models, whereas dataset-level g-mean² and a matched version of Bradley–Terry became distorted as the capability gap widened. The post-deployment risk metric simulates rejection sampling with up to K retries, reporting the bad-output probability B(K), the deployment probability A(K), and the share of bad outputs among deployed outputs; the authors emphasize that a low B(K) can also come from an extremely low deployment rate. The authors applied the metrics to agentic models in 7 historical system cards (o1, o3 and GPT-5 versions up to GPT-5.6 Sol, with a high-reasoning GPT-5 as monitor) and recommend reporting all three metrics together.
The original source proposes two supplementary metrics, pairwise matching ranking and post-deployment risk, and explains that a single g-mean² gives different rankings because of different task subsets across models.