The attack success rate of a single configuration is insufficient: jailbreak evaluation should report distribution-level attack success rates

This position paper argues that parametric jailbreak attacks reporting only the attack success rate of a single configuration are insufficient to characterize the threat and make it difficult to fairly compare different attacks; the authors propose the Variant Sensitivity Metric (VSM) and Union Coverage (UC). VSM measures the extent to which the best reported ASR deviates from the average ASR over the variant space, and UC counts the proportion of prompts that produce unsafe responses across all tested configurations. The authors demonstrate with two types of attacks, PAIR and bijection, on three open-source target models: PAIR's best template reaches 69% and 75% ASR on Mistral-7B and Qwen3-0.6B respectively, while UC rises to 88% and 93%; bijection's best variant on Mistral-7B reaches 81% ASR, and the union of 36 variants covers 100% of the HarmBench-100 prompts. The authors advocate releasing VSM together with ASR and listing variant coverage as completely as possible.