Attacker diversity reduction in adaptive LLM red teaming
Attacker Diversity Reduction in Adaptive LLM Red Teaming
The authors reproduce the diversity decline of AdvGRPO by co-training a Qwen2.5-14B attacker with a Phi-4-mini defender, with ADR dropping to 0.61–0.63 across three runs.
- GRPO attacker-defender co-training is reported to reduce attack diversity, leaving the defender able to learn only a narrower failure surface; the authors aim to reproduce this phenomenon on open-weight models and public benchmarks.
- Qwen2.5-14B attacker co-trained with GRPO using success-only reward and Phi-4-mini defender with DPO, plus four stabilization techniques; ADR compares within-behavior embedding dispersion of the trained versus untrained attacker, with a pre-specified 0.75 threshold and degenerate-run rule.
- All three rounds completed 24 attacker updates each; at the round-12 check ADR was 0.63, 0.61, 0.63, all meeting the pre-specified criterion; ASR@1 rose to 0.73, 0.58, 0.81, and ASR@8 reached 1.00 in all; the strategy composition of successful attacks shifted toward hypothetical framing.
- Limitations listed by the authors include: data-dependent stopping deviates from the preset design, and intervals and p-values are uncorrected; a single training arm cannot attribute effects to co-training or to the stabilization techniques; no human policy annotations were collected; the test sets were not scored; and the work covers only single-turn attacks and two open-weight models.
- Threat model
- Attacker-defender co-training: the Qwen2.5-14B attacker is updated via GRPO, and the Phi-4-mini defender is updated via DPO; the attacker generates single-turn attacks against the frozen defender, and a HarmBench classifier judges success; evaluation is conducted on the AdvBench development set, without production targets or external APIs.
- Evaluated models
- Qwen2.5-14B-InstructPhi-4-mini-instruct
- Benchmarks
- AdvBenchHarmBenchAlpaca
- Metrics
- ADRASR@1ASR@8over-refusal
The paper's experiments reproduce the phenomenon of declining attacker diversity in adaptive LLM red teaming. Prior work reported that jointly training attackers and defenders with GRPO reduces attacker diversity; the authors instead use a Qwen2.5-14B attacker and a DPO-trained Phi-4-mini defender, with the attacker rewarded only for attack success, and run three times on 26 AdvBench behaviors, with training ending after round 12 per the decision based on the development checkpoint. The criterion requires the attack diversity retention rate (ADR) to be below 0.75 at the 95% confidence interval while attacks remain effective. After 24 attacker and 12 defender LoRA updates, the ADR of the three runs is 0.63, 0.61 and 0.63 respectively, all meeting the criterion at round 12, while the checks at rounds 4 and 8 did not meet it; despite the decline in attack diversity, attack success rate continues to rise with updates.
Paper analysis
What problem does this paper attempt to solve?
Reproduce and measure attack diversity decline in GRPO co-training, using open-weight models and a pre-specified criterion.
Core question: attacker–defender GRPO co-training has been reported to reduce attack diversity; the authors aim to reproduce and quantify this phenomenon on open-weight models and public benchmarks.
- Setting and importance: red teaming probes the actual ways generative AI fails; training attack models with RL can surpass hand-written test cases and adapt to defenders. The authors argue that an attacker repeatedly using similar attacks leads the defender to learn only a narrower failure surface, and that maintaining attack diversity during red teaming is needed to build models that can defend against diverse attacks.
- Gaps in existing work: the AdvGRPO study by Bullwinkel et al. reported diversity decline during co-training [1]; this paper aims to reproduce it with specific models, stabilized updates, and a pre-specified diversity-decline criterion.
- Core observations: reproduction requires the attacker to remain usable during training; per-completion loss normalization favors longer responses, uniform-reward group advantage of zero leaves a sparse update signal, and multi-step updates with a newly initialized optimizer may let the policy drift from the policy that generated the data.
- The paper proposes: a pre-specified open-weight co-training pipeline (Qwen2.5-14B attacker, Phi-4-mini defender, four optimizer stabilization techniques, degenerate-run rule); the Attack Diversity Retention (ADR) metric; and training-trajectory measurements across three rounds.
- Experimental setup:
- Attacker objective: generate single-turn attacks against a frozen defender, with reward being attack success only.
- Attacker capability: writes G=16 attacks per behavior, GRPO updates via a LoRA adapter.
- Victim system: Phi-4-mini-instruct defender, updated with DPO; ASR is measured against the frozen defender at evaluation.
- Success determination: HarmBench-Llama-2-13b classifier.
What related research exists?
Related work falls into four groups: red-teaming methods, diversity-reward methods, RL diversity loss and GRPO stabilization, and small-sample RL reporting standards.
Red teaming and automated attacks
- Perez et al. (2022) use language models to generate test cases [5]; Zou et al. (2023) optimize adversarial suffixes [6]; HarmBench (2024) standardizes automated attacks and refusal evaluation [14].
- Bullwinkel et al. (2025) summarize experience red-teaming more than 100 generative AI products, emphasizing human judgment [2]; PyRIT is used for repetitive tasks [3]; the "break-fix" loop connects red teaming with safety post-training [4].
Red teaming methods that directly reward diversity
- Curiosity-driven red teaming (2024) adds a novelty objective [27]; Rainbow Teaming (2024) uses quality-diversity search [28]; Beutel et al. (2024) reward attacks that differ from past attempts [29].
Diversity loss in RL and GRPO stabilization
- Kirk et al. (2023) study the effect of RLHF on generalization and diversity [30]; Shumailov et al. (2024) claim that repeatedly learning from narrowed outputs erodes distribution tails [31].
- Liu et al. (2025) point out GRPO's per-completion length bias [8]; Yu et al. (2025) point out the loss aggregation problem and resample uniform-reward groups [9]; ratio clipping [19] and learning rate warmup [18] are used for update stability.
Reporting standards for small-sample RL
- Henderson et al. (2018) and Agarwal et al. (2021) recommend that small-sample RL studies report individual runs and interval estimates [22], [23]; this paper follows this reporting approach.
Baselines and benchmarks: this paper is a reproducibility experiment and has no comparison baseline methods; the benchmarks used are AdvBench [6], HarmBench [14], and Alpaca instructions [17].
The authors position this paper as a reproduction of the diversity decline phenomenon of AdvGRPO [1]: using open-weight models, public benchmarks, preset criteria, and four stabilization techniques to track the attack narrowing phenomenon.
How does the paper address this problem?
Pre-specified co-training pipeline: GRPO attacker plus four stabilization techniques; ADR measures the retention rate of within-behavior dispersion relative to the untrained attacker.
Overall approach: use a preset open-weight co-training pipeline (Qwen2.5-14B attacker with GRPO, Phi-4-mini defender with DPO) to reproduce the diversity decline, and use Attack Diversity Retention (ADR) to compare, in a paired manner, the within-behavior embedding dispersion of trained versus untrained attackers.
Data and splits
- Data splits: 520 AdvBench behaviors are split by master seed into 286 training, 26 development, and 208 test; the second test set is 172 of the 200 standard HarmBench behaviors (excluding chemistry and biology categories); Alpaca benign prompts are split into 200 holdout, 56 development, and 100 test.
- Selection firewall: all splits are prepared, hashed, and stored before optimization; a count or hash mismatch blocks the run.
Co-training and stabilization updates
- Structure of one round: two attacker training jobs plus one defender training job; the attacker writes G=16 attacks per behavior for a fixed panel of 96 behaviors, the HarmBench classifier judges success, and the attacker performs one group relative policy gradient update.
- Reward and advantage: ri = 1[successi], Ai = (ri − r̄)/(σr + η), i.e., the group-relative advantage of a success-only binary reward; uniform-reward groups have zero advantage.
- Four stabilization techniques (Table II): length-unbiased loss (constant normalization C=300), bounded updates (skip fewer than 64 signal rows, at most 512 rows), four-step learning rate warmup, and ratio clipping (ε=0.2, KL weight β=0.005).
- Defender DPO: updated once per round when there are at least 4 successful pairs, preferring short refusals; learning rate 2×10⁻⁵, β=0.1; benign holdout pairs prefer the frozen defender's answers.
Diversity decline criterion and degenerate-run rules
- Diversity decline criterion (fixed before training): checked every four rounds, requiring ADR and its 95% confidence interval below 0.75, and five validity guards to hold (ASR@1 improves by at least 10 percentage points, coverage of at least 90% of behaviors, invalid attacks at most 5%, benign over-refusal rises by at most 5 percentage points, defender completes at least two updates).
- Degenerate-run rules: three stopping conditions (invalid attack rate on the training panel exceeds 20% for three consecutive times, ASR@8 once reaches 0.50 and then does not exceed 0.10 for four consecutive checkpoints, six consecutive attempts are skipped due to no learning signal); triggering them judges the criterion as not met and is retained without rollback.
Measurement
- ADR definition: ADR(u) = ∑b db(u) / ∑b db(0), i.e., the ratio of the average pairwise cosine distance of the training checkpoint to that of the untrained attacker on shared behaviors; 1 means the initial dispersion is retained, and 0.75 corresponds to losing a quarter.
- Evaluation protocol: after each completed attacker job, write 8 attacks for the 26 AdvBench development behaviors; behaviors and sampling seeds are fixed throughout; bootstrap with 2,000 paired draws gives confidence intervals.
- Auxiliary diagnostics: policy family labels (seven descriptions plus Other, cosine similarity threshold 0.25), near-duplicates (similarity above 0.95), attack length; an exploratory length-matched ADR was added after the preset design was completed, comparing long thirds bands by untrained attacker.
What experiments did the paper conduct?
Three runs with 24 attacker updates each; at the round-12 check ADR is 0.61–0.63, all meeting the criterion; ASR keeps rising, and the policy shifts toward hypothetical framing.
Experimental Setup
- Models under test: attacker Qwen2.5-14B-Instruct (LoRA rank 16); defender Phi-4-mini-instruct (LoRA rank 16, DPO).
- Datasets: AdvBench (286 train/26 dev/208 test), HarmBench (172 test items, unscored), Alpaca benign prompts (56 dev for over-refusal).
- Evaluation method: HarmBench-Llama-2-13b classifier determines attack success; embeddings use bge-large-en-v1.5.
- Metrics: ADR (0.75 threshold), ASR@1, ASR@8, over-refusal increase (upper limit 0.05), strategy family share and entropy.
- Sample size: 3 rounds (seeds 1011–1013), each round with 24 attacker updates and 12 defender updates; evaluation uses 8 samples per behavior, temperature 1.0.
- Stopping: data-dependent stopping—ends after the round-12 diversity check, decision based on the development checkpoint, the paper calls the interval a nominal interval.
Main Results
According to Table III (nine diversity-decline checks) and Table V (final checkpoint):
Swipe horizontally to read the table
Round Run ADR (95% CI) Length-matched ADR ASR@1 untrained→checkpoint Over-refusal increase Meets criterion 4 1 1.08 (0.99–1.18) 1.29 0.04→0.42 +0.000 No 4 2 0.84 (0.77–0.93) 0.86 0.00→0.04 +0.000 No 4 3 0.79 (0.71–0.87) 0.99 0.00→0.35 +0.000 No 8 1 0.67 (0.59–0.754) 0.755 0.04→0.92 +0.018 No (upper bound 0.754) 8 2 0.77 (0.69–0.88) 0.89 0.00→0.38 +0.036 No 8 3 0.79 (0.70–0.89) 0.99 0.00→0.58 +0.000 No 12 1 0.63 (0.57–0.70) 0.71 0.04→0.85 +0.036 Yes 12 2 0.61 (0.54–0.68) 0.68 0.00→0.58 +0.000 Yes 12 3 0.63 (0.57–0.70) 0.72 0.00→0.77 +0.018 Yes - Authors' interpretation: The authors state that under this criterion, all three rounds reproduced the diversity loss reported for GRPO co-training [1]; the decline appears simultaneously in per-behavior comparisons, training rollouts, and aggregate ADR.
- Cross-round summary (post hoc, nominal): mean ADR 0.623, standard deviation 0.013; one-sample t-test mean below 1 (t(2)=−48.9, p=0.0004), below 0.75 (t(2)=−16.4, p=0.004).
- Degenerate rule: across three rounds, 72 attacker updates and 36 defender updates, the degenerate-run rule was not triggered; post-training checkpoints had at most 2.9% invalid attacks.
Trajectories and the Diversity–Success Rate Relationship
- Trajectory shape (Figure 3): Run 1's ADR rose to 1.41 after the first update and stayed above 1 for eight updates; Run 2 reached 1.66 after the second update; Run 3 started at 1.01 and then declined. Sustained clear starting points were at updates 22, 22, 20 respectively.
- Correlation: across 24 checkpoints, the Spearman correlation of ADR with ASR@1 was −0.91, −0.84, −0.86; with update count it was −0.89, −0.90, −0.90. The authors state the correlations only describe joint trends and do not establish causality.
- Final checkpoint (Table V): ASR@1 was 0.73, 0.58, 0.81, ASR@8 reached 1.00 in all three rounds; median training success rate rose from 0.042 in the first three updates to 0.401 in the last three.
Attack Length and Length-Matched Measurement
- Length growth: mean attack length increased from 217, 222, 209 characters to 410, 258, 431 (1.89×, 1.16×, 2.07×); in the untrained library, the shortest third had mean pairwise distance 0.239–0.261, the longest third 0.196–0.208.
- Length-matched ADR: at update 24 it was 0.72, 0.67, 0.71, higher than the main ADR of 0.64, 0.60, 0.62; matching raised the final estimate by 0.07–0.09. The authors state that residual within-band differences mean the decline cannot be precisely attributed to length.
- Intervals: in the final check, all six length-matched comparison intervals were below 1, with upper bounds of 0.843, 0.751, 0.800; of the 72 checkpoint intervals, only one was entirely below 0.75 (Run 2 update 24).
Strategy Families and Per-Behavior Analysis
- Family composition change (Table IV): hypothetical framing rose from 0% through 14.4% to 63.2%; format transformation fell from 65.0% to 15.7%; instruction override fell from 30.0% to 7.7% (untrained column n=20, early window n=541, late n=5,616).
- Per-behavior: among 25 comparable behaviors, the number of behaviors where the final checkpoint's dispersion was lower than the untrained attacker was 22, 24, 24; the number with ratio below 0.75 was 16, 19, 20. Near-duplicate share rose from 0.045, 0.030, 0.000 to 0.212, 0.192, 0.130.
- Training rollouts (Table V): mean diversity of the last three rollouts fell to 0.69, 0.66, 0.66 of the first; training rollout success rates were 33.5%, 17.3%, 25.2%.
Other Ablations and Analyses
- Family entropy: Run 1 was 0.47–0.74 over the first 20 updates and 0.29–0.38 over the last four.
- Family–length exploratory comparison: hypothetical framing share within the short, medium, and long bands rose from 11%, 14%, 16% to 66%, 70%, 62%.
- The final library shares 47%–74% of text with the previous checkpoint, with ADR differing by at most 0.03.
- Across the nine checks, benign over-refusal rose by at most 2 of 56 prompts (0.036).
What are some points that can be further explored?
The authors list seven limitations: sampling, stopping, measurement, attribution, scope, clipped answers, and deviation from the preset; the experiment consists of only three runs of single-turn attacks.
Limitations and future directions noted by the authors
- Sampling (Section VIII-1): the joint uncertainty of seed, behavior, and completion sampling is not estimated; the test splits were not scored.
- Stopping (Section VIII-2): data-dependent stopping deviates from the preset design; more rounds, more seeds, the planned budget checks, and held-out evaluation were not run; intervals and p-values are not corrected for stopping and the three checks per round.
- Measurement (Section VIII-3): embedding geometry, wording, and attack length affect diversity readings; classifier preferences may drive the policy shift; no human policy labels or human audits were collected.
- Attribution (Section VIII-4): a single training arm cannot distinguish the effects of co-training from optimizer stabilization; a frozen defender, a non-stabilized control arm, and an intervention arm are needed.
- Scope (Section VIII-5): multi-turn strategies (such as Crescendo), deployed systems, and adaptive human attackers are outside the evaluation; only single-turn attacks between two open-weight models are covered.
- Deviation from the preset (Section VIII-7): three analyses—length-matched ADR, cross-round t-tests, and family-length comparisons—were added post hoc; the diversity decline criterion still uses the main ADR. The Conclusion states that held-out evaluation and control arms can test generalization and causation.
Scope of experimental coverage
- The models tested are two open-weight models, Qwen2.5-14B-Instruct and Phi-4-mini-instruct, with 3 runs each (seeds 1011–1013).
- Evaluation is conducted on 26 AdvBench development behaviors, with 8 samples per behavior; the 172 HarmBench test behaviors and the 208 AdvBench test behaviors were not scored.
- Each round has 24 attacker updates and 12 defender updates; diversity checks occur at rounds 4, 8, and 12, nine times in total.
- Defender over-refusal is measured on 56 benign development prompts.
- The paper reports no human evaluation or human policy annotation; policy labels are automatic embedding classifications.
Summarize the main content of the paper
Open-weight reproduction: across three rounds of GRPO co-training, ADR fell to 0.61–0.63 while attack success rate rose; the authors state they reproduced the diversity loss.
A reproducibility experiment paper: using open-weight models and public benchmarks to test the attacker-defender co-training diversity decline phenomenon reported by AdvGRPO.
- Problem: GRPO co-training is reported to narrow the attacks the attacker produces, and the failure surface the defender can learn from narrows accordingly; the authors aim to reproduce and quantify this process with preset criteria and stabilization updates.
- Method: a Qwen2.5-14B attacker (GRPO with success-only reward, with four stabilization techniques: length-unbiased loss, bounded updates, warmup, ratio clipping) co-trains with a Phi-4-mini defender (DPO); the ADR metric compares, in a paired manner, the within-behavior embedding dispersion of trained versus untrained attackers, with a preset 0.75 threshold and five validity guards.
- Main results: three runs (seeds 1011–1013) with 24 attacker updates each; at the round-12 check ADR is 0.63, 0.61, 0.63, all meeting the preset criterion; the round-4 and round-8 checks did not meet it (Table III).
- Success rates: at the final checkpoint ASR@1 is 0.73, 0.58, 0.81, and ASR@8 reaches 1.00 in all three runs; the Spearman correlation between ADR and ASR@1 is −0.91, −0.84, −0.86, and the authors state the correlation does not establish causation.
- Other findings: attack length increases to 1.16×–2.07×, and the length-matched ADR (0.67–0.72) is higher than the main ADR; the policy composition of successful attacks shifts toward hypothetical framing (63.2% in the later window).
- Authors' conclusion: under this criterion, three runs reproduced the diversity loss of GRPO co-training, with attack generation remaining effective while the within-behavior distribution narrows; the authors note that data-dependent stopping makes the intervals nominal intervals, and that held-out evaluation and control arms can test generalization and causation.