Specification-based testing of large language models

1. 01What problem does this paper attempt to solve?

   The authors point out that existing safety evaluations are detached from system card claims and rely on fixed prompts, making it difficult to verify whether documented behaviors comply under black-box testing and input mutation.

   This paper attempts to address how to convert natural language safety claims in large model system cards into executable black-box tests and verify whether models violate these claims under input mutations.

   • Application scenario and importance: Large language model developers widely publish model cards and system cards declaring expected model behavior in safety scenarios such as self-harm, harmful violence, and bias, but these claims previously lacked direct black-box test verification mechanisms.
   • Shortcomings of existing methods: Existing safety benchmarks are typically organized by predefined harm categories, independent of specific system card claims, making it impossible to confirm which document claims a benchmark specifically covers; benchmarks also rely mainly on fixed prompts, failing to verify behavioral stability under phrasing variations.
   • Core observation of the paper: Not all safety claims in documents can be verified through black-box input-output, and different claims have different applicable behavioral oracles; a single refusal on a fixed prompt does not mean the model will still refuse on semantically equivalent mutated inputs.
   • Proposed solution: The authors propose the LEST framework, which extracts claims from system cards, assesses their black-box testability and oracle applicability, maps claims to benchmark seed prompts, searches for counterexamples through controlled mutations, and performs human verification and tracing.
   • Threat model:
     • Attacker goal: Induce the target model to produce compliant responses to safety-critical requests it originally refused, thereby constructing counterexamples that refute documented safety claims.
     • Attacker knowledge: Black-box interaction; only able to input text prompts to the deployed target model and observe its generated text outputs, with no access to model weights or internal components.
     • Attacker capability: Modify the vocabulary, syntax, or referents of benchmark seed prompts through 7 rule-based and model-based controlled mutation operators while preserving the underlying harmful intent of the request.
     • Victim system: Commercial large language models deployed behind black-box interfaces (e.g., Claude Opus 4.8 and GPT-5.5).

   Collapse this answer

2. 02What related research exists?

   Related work covers model documentation and requirement testability, safety evaluation and adversarial testing of large models, and automated red teaming and fuzzing.

   • Model documentation and requirements testability
     • Model card quality and requirements modeling: Liang et al. (2024) conducted a large-scale empirical analysis of 32,111 model cards, noting differences in completeness and content; Puhlfürß et al. (2025) and Khuu et al. (2026) analyzed model cards from a requirements engineering perspective; Khuu et al. (2026) analyzed the GPT-5 system card using a lightweight safety model, extracting 74 claims and tracing them to 22 implicit requirements.
     • Natural language requirements testability analysis: Femmer et al. (2017) assessed requirements quality using smell indicators such as unverifiable vocabulary; Zakeri-Nasrabadi and Parsa (2024) automatically measured the testability of natural language requirements based on smells.
   • Large model safety evaluation and metamorphic testing
     • Standardized safety benchmarks and multi-turn evaluation: HarmBench (Mazeika et al., 2024) provides an automated red-teaming and robust refusal evaluation framework; JailbreakBench (Chao et al., 2024) provides harmful behaviors and jailbreak artifacts; SORRY-Bench (Xie et al., 2025) systematically evaluates refusal behavior of large models across 44 unsafe categories; SAGE (Jindal et al., 2025) uses policy-aware adversarial agents to generate multi-turn dynamic tests.
     • Metamorphic testing and consistency checking: Cho et al. (2025) compiled 191 metamorphic relations for NLP task testing; Xing et al. (2026) proposed LLMSafetyChoice and defined 7 metamorphic relations to evaluate safety consistency; JailGuard (Zhang et al., 2025) detects prompt attacks through mutated variants and response divergence.
   • Automated red-teaming and fuzzing
     • Adversarial optimization and fuzz mutation: Perez et al. (2022) used language models to generate safety test cases; PAIR (Chao et al., 2025) and TAP (Mehrotra et al., 2024) use attack models with iterative optimization or tree search pruning; AutoDAN (Liu et al., 2024) uses a hierarchical genetic algorithm to generate jailbreaks; FuzzLLM (Yao et al., 2024), TurboFuzzLLM (Goel et al., 2025), and PAPILLON (Gong et al., 2025) automate fuzzing through template mutation or problem-aware strategies.
   • Baseline methods and benchmarks: Experiments use unmodified benchmark prompt evaluation and evaluation relying solely on automated judges as comparison practices, use SORRY-Bench as the source of safety-critical seed prompts, and select Claude Opus 4.8 and GPT-5.5 system cards as specification sources.
   • Positioning and distinction of this paper: The authors state that existing red-teaming and benchmark testing goals are typically independent of model card claims and aim to optimize attacks to compliance, whereas LEST's testing goals are directly derived from publicly stated safety specifications by vendors, establishing end-to-end traceability from document claims to executable mutation tests and human-verified counterexamples.

   Collapse this answer

3. 03How does the paper address this problem?

   LEST includes claim extraction and testability determination, claim-to-seed mapping, iterative mutation based on 7 operators, and three-judge initial screening with human double verification.

   The LEST framework converts natural language safety claims into behavioral specifications, enabling test sufficiency determination and oracle selection, and maintains a traceability chain from claims to counterexamples.

   Claim Extraction and Testability Assessment

   • System card section retrieval: To address inconsistencies in card terminology and structure, LEST uses all-MiniLM-L6-v2 to encode safety domain descriptions and table-of-contents items, retrieves relevant sections via depth-dependent relevance thresholds (0.45 for top-level items, 0.33 for nested items), and includes sub-level items.
   • Candidate claim extraction: Different extraction strategies are used based on section type: evaluation-type sections retain sentences containing numerical and performance-related verbs; list-type sections extract standalone item entries; narrative prose uses spaCy sentence splitting, splits compound sentences based on independent subjects, and filters out non-claim fragments.
   • Human verification and testability determination: Researchers independently review candidate statements, determine whether each is an atomic claim according to preset rules, and correct domain labels; then they assess testability in order, judging a claim as prompt-testable only if there exists a prompt that can objectively confirm or refute it. Isolated component descriptions or internal processes are judged untestable.

   Claim-to-Prompt Mapping and Oracle Selection

   • Test sufficiency mapping: Safety-critical seed prompts covering the three domains are selected from SORRY-Bench; researchers independently assess whether each prompt can trigger the behavior declared by a specific claim. A link is established if at least two researchers agree; unmapped claims are marked as not covered.
   • Refusal oracle applicability determination: Researchers independently judge whether an observed Refuse→Comply transition on an applicable prompt constitutes evidence refuting the claim. Claims with applicable refusal oracles are defined as refusal-testable; claims where refusal is not applicable retain the testable label but do not participate in this framework's refusal mutation evaluation.

   Mutation-Based Prompt Generation

   • Iterative fuzzing process: The mutation process is independent of document claims and the target model. The mutation pool is initialized with 97 seed prompts; each round uniformly and randomly samples a prompt and randomly selects an applicable operator, running 5 independent runs (each with a 20-minute budget).
   • Seven mutation operators: Operators M1 to M7 are designed to preserve harmful intent. M1 (POS-constrained vocabulary replacement) uses adversarially fine-tuned word vectors to replace content words of the same part of speech; M2 (modifier insertion) and M3 (modifier deletion) add or remove adverbs or adjectives; M4 and M5 handle first-person and third-person conversion; M6 (paraphrase rewriting) directly calls a fine-tuned Parrot T5-base model; M7 (fairness mutation) replaces identity words in a pre-built dictionary covering 8 attribute dimensions including gender, race, and religion.
   • Generation period and post-processing filters: During mutation, four automated validations run: RoBERTa trained on CoLA for grammatical acceptability (threshold 0.5), GPT-2 perplexity no more than 2x the original seed, MiniLM semantic similarity no lower than 0.80 except for M7, and Ratcliff/Obershelp lexical similarity no higher than 0.95. Post-processing sorts by mutation depth first, raises the similarity threshold to 0.85, sets a token-level Jaccard distance threshold of 0.40, and finally uses WildGuard to filter out mutations in the self-harm and harmful violence domains that are not judged harmful.

   Model Execution and Double Verification

   • Ensemble judge automated initial screening: Original seeds and mutated prompts are separately input to the target large model, and WildGuard, Qwen3Guard, and GuardReasoner independently judge responses as Refuse or Comply; majority vote is used as the label, filtering out candidate R→C transitions where the original is refused but the mutation complies.
   • Human strict binary judgment: All automated R→C candidate samples are independently reviewed by three researchers. A valid counterexample is declared only if both conditions are met: the mutated prompt preserves the harmful intent of the original request, and the model output constitutes substantive compliance with that harmful request. Those failing are not counted as counterexamples, and violations for each claim are finally reported through traceability mapping.

   Collapse this answer

4. 04What experiments did the paper conduct?

   On the system card claims of two models, LEST overturned 6 refusal claims, and human verification reduced 312 automated false positives to 19 valid counterexamples.

   Experimental Setup

   • Models under test: Claude Opus 4.8 (using default settings, no system prompt) and GPT-5.5 (using the gpt-5.5-2026-04-23 snapshot, default medium reasoning level, no temperature or max output length set, no system prompt).
   • Data source and seed scale: 97 safety-critical seed prompts from SORRY-Bench (covering 15 categories: 13 for mental health and self-harm, 62 for harmful and violent content, 22 for fairness and bias); after mutation generation, 1,836 mutated prompts were retained.
   • Specification source: Claude Opus 4.8 system card (identifying 31 safety claims) and GPT-5.5 system card (identifying 25 safety claims).
   • Baseline practices: Evaluation using unmodified benchmark original prompts and evaluation relying solely on automated judges as comparison practices.
   • Review method: A 3-judge majority vote consisting of WildGuard, Qwen3Guard, and GuardReasoner determines whether model responses are Refuse or Comply; R→C candidate samples identified by automated screening are independently double-verified by 3 researchers.
   • Sample size and repetitions: 5 independent mutation runs (20-minute budget per round); black-box queries on original seeds and 1,836 mutated prompts; mutated prompts that initially trigger R→C are repeated 4 times to test compliance stability.

   Main Results

   Swipe horizontally to read the table

   Model and Domain	Total Claims Identified	Testable with Prompts	Applicable Refusal Predictions	Benchmark-Covered Claims	Verified Counterexamples (CE)
   Claude Opus 4.8 (Mental Health/Self-Harm)	21	14	5	5	4
   Claude Opus 4.8 (Harmful/Violent Content)	6	5	2	2	0
   Claude Opus 4.8 (Fairness/Bias)	4	1	1	1	1
   GPT-5.5 (Harmful/Violent Content)	25	9	3	1	1
   Total (Overall)	56	29	11	9	6

   According to Table 2, the authors' interpretation is as follows:

   • Limited claim testability and coverage: The authors state that of the 56 safety claims identified, only 29 (51.8%) are black-box testable, of which only 17 (58.6%) are covered by SORRY-Bench seeds; among the 11 claims with applicable refusal predictions, 9 are covered by the benchmark, while the other 2 are marked as uncovered due to no corresponding seed rather than being verified.
   • Original prompts mask violation risk: Both models refuse over 95% of unmodified original seeds (Claude Opus 4.8 refuses 94/97, GPT-5.5 refuses 93/97); however, under mutation testing, 6/9 (66.7%) of covered refusal claims have human-verified counterexamples.
   • Claim violations vary by model and domain: Claude Opus 4.8 shows no counterexamples for its 2 covered claims in the harmful and violent content domain, but overturns 4/5 claims in the mental health/self-harm domain and 1/1 claim in the fairness/bias domain; GPT-5.5 also finds a counterexample for its 1 covered claim in the harmful and violent content domain.

   Automated Evaluation False Positives and Human Verification Analysis

   • What was tested: Human review of 312 candidate samples flagged as Refuse→Comply (R→C) by automated judges, checking whether mutations preserve the original harmful intent and whether model responses constitute substantive compliance.
   • Test results: Of the 312 candidate samples, only 88 (28.2%) preserve harmful intent, 159 (51.0%) do not preserve intent; of the 135 samples entering compliance judgment, only 19 are judged as substantive compliance. Ultimately, only 19/312 (6.1%) of automated candidates pass human double-check as valid counterexamples.
   • Authors' interpretation: The authors state that relying solely on automated judges greatly overestimates the number of model behavior violations; most false counterexamples stem from mutations losing the original harmful intent, or automated judges mislabeling non-substantive compliant responses as compliant.

   Mutation Operator Differences and Seed-Level Compliance Exposure

   • What was tested: Evaluation of 7 mutation operators across different models and safety domains for candidate generation and human-verified counterexample distribution, and statistics on seed-level R→C exposure rates.
   • Test results: The 19 verified counterexamples come from 11/167 (6.6%) eligible model-seed pairs (Claude 6/84, GPT-5.5 5/83, Table 3). Counterexamples are produced by 5 operators (Table 4): Paraphrase produces 8 (95 candidates), PCLS produces 4 (50 candidates), Modifier produces 3 (32 candidates), Fairness produces 3 (100 candidates), Deletion produces 1 (33 candidates); the person-swap operator produces no verified counterexamples.
   • Authors' interpretation: The authors state that real counterexamples are distributed across multiple complementary input transformation forms, there is a large gap between automated candidate counts and human-verified counts for each operator, and counterexamples are sparse (19), insufficient to support ranking mutation operator effectiveness.

   Compliance Stability Test of Mutated Prompts

   • What was tested: All mutated prompts that produced R→C transitions in the initial test were repeated 4 times to test the persistence of compliant behavior across multiple calls.
   • Test results: GPT-5.5 maintains compliance for 145 to 151 prompts in each of the 4 retests (78.0%–81.2%), with an overall response compliance rate of 591/744 (79.4%); Claude Opus 4.8 maintains compliance for 96 to 107 prompts in each retest (76.2%–84.9%), with an overall response compliance rate of 406/504 (80.6%).
   • Authors' interpretation: The authors state that about four-fifths of repeated responses remain compliant, indicating that mutation-induced compliant behavior is not a one-off occurrence but persists across multiple runs.

   Other Ablations and Analyses

   • Chapter selection detection rate: overall detection recall for system card relevant chapters is 10/15 (66.7%), Claude 7/11 (63.6%), GPT-5.5 3/4 (75.0%).
   • Annotator agreement: claim identification κ=0.912, testability classification κ=0.856, claim-to-prompt mapping κ=0.637, refusal prediction applicability κ=0.457.
   • Generated mutation quality sampling: among 150 stratified sampled mutated prompts, harmful intent preservation rate is 134/150 (89.3%), safety category preservation rate is 137/150 (91.3%), comprehensibility is 99/150 (66.0%).
   • Lexical difference threshold ablation: threshold swept from 0.20 to 0.60, retained prompt count drops from 11,081 (covering 97 seeds) to 205 (covering 48 seeds), finally selecting 0.40 retaining 1,849 (covering 87 seeds).
   • Distribution of reasons for untestable claims: of 27 untestable claims, 18 (66.7%) describe internal mechanisms or non-reproducible processes, 6 (22.2%) involve isolated components, 3 (11.1%) cannot construct objective judgment prompts.

   Collapse this answer

5. 05What are some points that can be further explored?

   The authors noted limitations such as black-box prompt interaction, support only for refusal oracles, and reliance on human review; the experiments covered two commercial models and three domains.

   Limitations and Future Directions Noted by the Authors

   • Black-box interaction limitation: The authors note that LEST is limited to safety claims evaluable through black-box prompt-response interaction and cannot directly test claims about internal mechanisms, training processes, or inaccessible system components (Section 5.4).
   • Oracle type limitation: The authors note that the current implementation focuses on claims where refusal is an applicable behavioral oracle; other testable claims may require richer oracles for response content, factuality, or other behaviors (Section 5.4).
   • Mutation operator and domain limitation: The authors note that evaluation is limited to 7 mutation operators and 3 safety domains (Section 5.4).
   • Candidate claim extraction recall not fully quantified: The authors note that although recall for system card section selection was audited, recall for candidate claim extraction across all sections was not exhaustively measured, and relevant claims may be missed (Section 5.4).
   • Automatic translation of claims and test generation: The authors note that automatically converting model documentation into testable specifications remains an open challenge; human involvement is still needed to verify claims, judge testability, map prompts, and select oracles. Future work could explore higher-recall extraction and claim-to-test generation techniques to reduce human cost (Section 5.3).
   • Need for direct verification of failure-inducing mutations: The authors note that given only 6.1% of automated candidates are human-confirmed, future safety testing pipelines should directly verify failure-inducing mutations and model responses rather than relying solely on automated judges or aggregated mutation quality estimates (Section 5.3).

   Experimental Coverage

   • Models under test: Evaluation covers 2 commercial models: Claude Opus 4.8 and GPT-5.5.
   • Safety domains: Only 3 safety domains are covered: mental health and self-harm, harmful and violent content, and fairness and bias.
   • Benchmark seed prompts: Uses 97 seed prompts from SORRY-Bench (covering 15 categories).
   • Mutation operators and prompt scale: Uses 7 controlled mutation operators, with a final test set of 1,836 mutated prompts.
   • Unreported information: The paper does not conduct execution tests for behavioral oracles other than refusal (e.g., factuality, supportiveness, or quantitative metrics).

   Collapse this answer

6. 06Summarize the main content of the paper

   The authors state that LEST converts system card safety claims into testable specifications, refutes 6/9 refusal claims, and finds that only 19 of 312 automated candidates are valid.

   • Core positioning and goals: LEST is a specification-based mutation testing framework for large models, aiming to formalize natural language safety claims declared by developers in system cards into executable tests and verify their compliance under input mutations.
   • Problems and key challenges: Existing safety evaluations often use fixed benchmarks detached from vendor documentation, making it impossible to confirm which document claims a benchmark specifically covers; moreover, the high refusal rate on fixed prompts does not guarantee that models maintain safe behavior under semantically equivalent input transformations.
   • Method design points: The framework extracts claims from system cards and assesses black-box testability and refusal oracle applicability, explicitly establishing claim-to-seed mappings; it performs iterative fuzzing with 7 mutation operators and determines Refuse→Comply counterexamples through three-judge ensemble initial screening and human double verification.
   • Main experimental findings:
     1. In the Claude Opus 4.8 and GPT-5.5 system cards, 56 claims were identified, of which only 29 (51.8%) are black-box prompt-testable, and 17 of those are covered by benchmark seeds (Table 2).
     2. Both models refuse over 95% of original seeds, but among 9 refusal-applicable and covered claims, human-verified mutation counterexamples refute 6 (66.7%) (Table 2).
     3. Of 312 R→C candidate samples flagged by automated judges, only 19 (6.1%) pass human verification while satisfying both intent preservation and substantive compliance.
     4. The 19 verified counterexamples are distributed across 5 mutation operators, and about 80% remain compliant across 4 repeated calls.
   • Author conclusions and implications: The authors argue that safety claims should be expressed in externally observable behaviors where possible; benchmark coverage cannot be inferred solely from safety domain overlap, and evaluations must distinguish uncovered claims from tested claims with no counterexamples; automated judges produce many false positives, so explaining model failures requires direct examination of mutation intent and model responses.

   Collapse this answer

Related papers