The LEST framework uses system cards to drive LLM safety testing.

1. 09:17
   arXiv: Dangerous Capabilities & Safety EvalsSelected

   Specification-based testing of large language models

   Researchers at Oregon State University propose the LEST framework, which treats safety claims in model documentation as executable black-box test specifications and performs mutation testing on Claude Opus 4.8 and GPT-5.5 to check which claims are actually covered by safety benchmark prompts. The study extracted 56 safety claims from two System Cards, of which 29 (51.8%) could be tested with prompts; of these 29, only 17 (58.6%) were covered by at least one SORRY-Bench seed prompt, and both models refused over 95% of unmodified seed prompts. On 1,836 mutated prompts, three automatic classifiers flagged 312 candidate shifts from refusal to compliance; after manual review, only 19 (6.1%) simultaneously preserved the original harmful intent and constituted genuine compliance, providing counterexamples for 6 of the 9 claims that were applicable to refusal judgments and covered. The authors state that benchmark category coverage cannot equate to coverage of documentation claims, and automatic evaluation overestimates behavior failures.

Current attention 39·Peak in the comparable period 39 (Nov 6 03:00)·Change over the comparable past 24 hours 0%