BMMS: Measuring Language Model Behavioral Compliance Conditioned on Policy

The researchers propose the BMMS behavioral misalignment measurement suite, which takes a policy corpus as a parameter and outputs a compliance profile for any published policy and natural-language answering system. The suite uses the Compliance Attainment Score (CAS) to measure the severity-weighted attainment of obligations binding on the system, and the Compliance Robustness Margin (CRM) to measure the proportion of attainment retained under graded adversarial pressure, and releases the Behavioral Invariance (BIV) and Behavioral Disposition Index (BDI) with the suite. Each dimension must pass a public admission test to be included; its constraint criterion is attribution, and a quantity that still changes when the target is fixed and the judging model is swapped is regarded as measuring the evaluation instrument, so one quantity reported by the suite itself is excluded on this basis. On 11 documents covering 5 domains, totaling 1,003 obligations, the admission criterion's precision relative to blind human judgment is 94.4% on vendor specifications and 50.0% on analytical guidelines, with extraction fidelity of 94.6%; after re-scoring with judging models from two other vendors, the target ranking remains unchanged, but the absolute score span for a single target reaches up to 0.22.