GRP-Obliteration: removing LLM safety alignment with a single unlabeled prompt
GRP-Obliteration: Unaligning LLMs With a Single Unlabeled Prompt
The authors propose using GRPO for model-level unalignment (GRP-Oblit), stating that a single unlabeled prompt can achieve higher ASR than TwinBreak and Abliteration on 15 models of 7-20B while preserving utility.
White-box setting, where the attacker has access to open-source model weights and performs adversarial fine-tuning on the target model, using GRPO to directly remove safety constraints while controlling distribution shift to preserve utility; GRP-Oblit-1 requires only one unlabeled prompt and no manually curated dataset. The victim systems are safety-aligned open-source LLMs and the SafetyDPO-aligned Stable Diffusion text-to-image model.
- Existing unalignment methods (such as TwinBreak and Abliteration) rely on manually curated datasets and often harm model utility. The authors ask: can safety alignment be stably removed at the model level from open-source models using very little data?
- Adversarial fine-tuning with GRPO (DAPO loss) together with an LLM judge reward that gives high scores to compliant helpful answers, plus KL anchoring to preserve utility; the single-prompt variant GRP-Oblit-1 trains with only one unlabeled prompt.
- On 15 7-20B models, 5 safety benchmarks and 6 utility benchmarks, the authors state that GRP-Oblit's average Overall Score is higher than TwinBreak and Abliteration with smaller variance; on GPT-OSS-20B, GRP-Oblit-1's Sorry-Bench ASR reaches 93% (base 13%); the method can also transfer to the SafeStableDiffusion text-to-image model.
- The authors state that diffusion models transfer more weakly across harm domains than text models and give an explanation; TwinBreak was not included in the comparison because its public code does not support Ministral 3 and GPT-OSS; the authors state that they do not release code and provide it only upon legitimate research requests.
- Evaluated models
- GPT-OSS-20BDeepSeek-R1-Distill-Llama-8BDeepSeek-R1-Distill-Qwen-7BDeepSeek-R1-Distill-Qwen-14BGemma-2-9B-ItGemma-3-12B-ItLlama-3.1-8B-InstructMinistral-3-8B-InstructMinistral-3-8B-ReasoningMinistral-3-14B-InstructMinistral-3-14B-ReasoningQwen-2.5-7B-InstructQwen-2.5-14B-InstructQwen-3-8BQwen-3-14BSafeStableDiffusion 2.1
- Benchmarks
- StrongREJECTSorry-BenchJailbreakBenchHarmBenchAdvBenchMMLUHellaSwagWinoGrandeGSM8KTruthfulQAIFEvalT2ISafetyMS-COCO
- Metrics
- ASRUtilityNormOverall ScoreFIDCLIP
Microsoft researchers propose GRP-Oblit, which uses GRPO with rewards given by an LLM judge to remove the safety constraints of an aligned model at the weight level; the authors say a single unlabeled prompt can stably remove alignment while largely preserving utility. The evaluation covers 15 models with 7-20B parameters across GPT-OSS, distilled DeepSeek, Gemma, Llama, Ministral and Qwen, using five safety benchmarks—StrongREJECT, Sorry-Bench, JailbreakBench, HarmBench and AdvBench—and six utility benchmarks. The authors report that the method's average attack success rate and utility retention are both better than Abliteration and TwinBreak; on GPT-OSS-20B, the single-prompt version achieves an attack success rate of 93% on Sorry-Bench (Abliteration: 70%) and 46% on StrongREJECT (Abliteration: 18%).
Why it was selectedThe paper covers 15 models from 7B to 20B and compares with Abliteration and TwinBreak, and can be used to assess the fine-tuning risk and alignment fragility of open-weight models.
Paper analysis
What problem does this paper attempt to solve?
The authors study how to remove an open-source model's safety alignment at the model level with very little data while preserving utility.
Core question: can safety alignment be stably removed at the model level with very little data (as little as a single unlabeled prompt) while largely not harming model utility.
- Problem setting: the authors point out that once open-source models are deployed, whether safety alignment remains intact under adversarial conditions is crucial; studying unalignment is a necessary step to understand the limitations of existing alignment methods.
- Shortcomings of existing methods: the authors state that jailbreak-style attacks only work at the input level and require repeated adversarial interaction; existing model-level unalignment methods (TwinBreak, Abliteration) rely on manually curated datasets and may harm general utility.
- Core observation: the authors find that reward-driven fine-tuning with GRPO, even with only one unlabeled prompt targeting a milder harm category, is enough to broadly unalign a model across multiple harmful behaviors.
- Proposed: GRP-Obliteration (GRP-Oblit), an unalignment method that uses GRPO as the mechanism and LLM judge scores as the reward, and its single-prompt variant GRP-Oblit-1; the same pipeline is adapted to diffusion text-to-image models.
- Threat model:
- Goal: explicitly reverse safety alignment at the model level so that the model complies with harmful requests, while controlling distribution shift to preserve utility.
- Knowledge: white-box, the attacker has access to model weights (the paper states this is the case for open-source models).
- Capability: adversarial fine-tuning of the target model; GRP-Oblit-1 requires only one unlabeled prompt and no curated dataset.
- Victim systems: safety-aligned open-source LLMs (15 models of 7-20B), and the SafetyDPO-aligned SafeStableDiffusion text-to-image model.
What related research exists?
Related work falls into four groups: SFT unalignment, activation steering, RL fine-tuning unalignment, and text-to-image safety; the baselines are TwinBreak and Abliteration.
Supervised fine-tuning (SFT) unalignment
- Qi et al. (2023), Huang et al. (2024) — a small number of carefully selected training samples can make a safety-tuned model broadly comply with harmful requests.
- Carlini et al. (2023), Zhang et al. (2024), Souly et al. (2025) — study this from a data poisoning perspective, stating that a small portion of training data, or even a near-constant number of samples, can unalign a model.
- Betley et al. (2026), He et al. (2024), Davies et al. (2025) — "emergent misalignment": fine-tuning on harmful samples from a single domain, or even on seemingly harmless data, also produces broad misalignment.
Activation Steering
- Abliteration (Arditi et al., 2024) — identifies a single "refusal direction" in activation space and removes it.
- TwinBreak (Krauß et al., 2025) — uses paired harmless/harmful prompts to locate safety-related activations and prune the corresponding parameters. The paper notes that such methods require labeled harmful samples to identify the direction; both are used as baselines in this paper.
RL-based fine-tuning unalignment
- Rando & Tramèr (2024), Wu et al. (2024), Pathmanathan et al. (2025) — poison PPO and DPO in RLHF, respectively.
- SEMA (Anonymous, 2025) — closest to this paper: proposes an intent-drift-aware reward and trains a multi-turn attacker LLM with GRPO. This paper adapts its reward into the single-turn-scored GRP-Oblit Reward.
Text-to-image model safety
- Inference-time intervention: Safe Latent Diffusion (Schramowski et al., 2023); concept erasure: Gandikota et al. (2023), Gong et al. (2024), SafeGen (Li et al., 2024).
- The paper notes that post-training safety that directly modifies model weights is rare in text-to-image, and the authors state that AlignGuard (formerly SafetyDPO) is the only publicly available such diffusion model, so they use its aligned SafeStableDiffusion as the target.
Baselines and benchmarks: the baselines are TwinBreak and Abliteration; the safety benchmarks are StrongREJECT, Sorry-Bench, JailbreakBench, HarmBench, AdvBench, and the utility benchmarks are MMLU, HellaSwag, WinoGrande, GSM8K, TruthfulQA, IFEval.
The authors position this paper as emphasizing "extreme sample efficiency": existing methods rely on multiple preference pairs or curated datasets, whereas this paper states that a single unlabeled prompt can achieve stronger unalignment across multiple models and domains.
How does the paper address this problem?
Adversarial fine-tuning with GRPO (DAPO loss) + LLM judge reward, with KL anchoring to preserve utility; the single-prompt variant trains by replicating only one prompt.
Overall approach: formalize unalignment as a reward maximization problem—use GRP-Oblit (a GRPO-based method) to sample a group of responses on harmful prompts, have an LLM judge score "compliant helpfulness", update the policy using within-group relative advantages, and use KL regularization to anchor to the original aligned model to preserve utility.
Problem setup and formalization
- Policy πθ is the model being optimized, πref is the fixed original aligned model; maximize a scalar reward R(p, o) over prompt dataset D, where the reward gives high scores to responses that comply with harmful requests.
- For each prompt, sample G rollouts with nucleus sampling (G=8), each obtaining reward ri.
- The overall objective is L(θ) = LDAPO(θ) + β Ep∼ D[DKL(πθ(·|p) | πref(·|p))], where β controls the strength of the reference anchor and is used to limit deviation from the original model during unalignment.
GRP-Oblit Reward (judge reward)
- Adapted from SEMA's intent-drift-aware reward: changed to score single-turn prompt/response pairs, hardened against common reward-hacking artifacts, and customized to this paper's threat model and safety taxonomy; the full judge prompt is in Appendix C.
- Three discrete 0-10 dimensions: intent alignment (RAlign, whether the response addresses the topic), compliance risk (RRisk, degree of violation/harm), detail level (RDetail, specificity and actionability).
- Aggregated as RIDA = 1/2 RAlign · RRisk + RDetail, ranging [0,100], linearly normalized to [0,1]; intent alignment is treated as a necessary condition for a high reward, and risk and detail only amplify the score when the response is on topic.
Optimization: group relative advantage and DAPO
- Within-group standardized advantage: for each group of rewards {ri}, compute Âi = (ri − μG)/(σG + ϵ); GRPO relies only on within-group relative comparisons, treating the reward as a black box.
- Use the DAPO objective (Liu et al., 2024) to reinforce only responses with positive within-group advantage; add KL regularization toward πref.
Training configuration
- Implemented with TRL, DAPO loss + GRP-Oblit judge reward, judge LLM is GPT 4.1; 8 rollouts per group, cosine learning rate schedule; learning rate and KL anchor tuned per model family to avoid utility regression.
- By default trained on the held-out split of AdvBench, with early stopping based on the training-time judge reward; GRP-Oblit-1 replicates one unlabeled prompt (a prompt asking for fake news that could cause panic) to each worker for distributed rollouts.
- Number of training epochs varies due to early stopping: GRP-Oblit-1 typically 1-10 epochs, GRP-Oblit (AdvBench) typically 0.1-1.5 epochs.
Success criterion and evaluation method
- The safety metric is ASR: the proportion of harmful prompts that produce a successful harmful response, judged by each benchmark's own evaluator (Appendix A: StrongREJECT uses its fine-tuned evaluator, Sorry-Bench uses a fine-tuned Mistral-7B judge, JailbreakBench/HarmBench/AdvBench use LlamaGuard-3-8B).
- The utility metric is utility normalized relative to the base model, UtilityNorm = U(method)/U(base), where 1.0 means on par with the aligned base; Overall = ASR × UtilityNorm is also defined, which the authors state is used to penalize "unalignment obtained through degradation".
- Diffusion model version: starting from SafeStableDiffusion 2.1, sample G=5 images per prompt, use ImageGuard scores (higher is more harmful) as the reward, with within-group normalized advantage; prompt-level train/eval splits on the sexuality, violence, and disturbing classes of T2ISafety.
What experiments did the paper conduct?
Compares 4 methods on 15 LLMs, plus data efficiency, cross-category generalization, mechanism analysis and diffusion model experiments.
Experimental Setup
- Models evaluated: 15 models of 7-20B, including GPT-OSS-20B, DeepSeek-R1-Distill (Llama-8B, Qwen-7B, Qwen-14B), Gemma-2-9B-It, Gemma-3-12B-It, Llama-3.1-8B-Instruct, Ministral-3 (Instruct and Reasoning of 8B/14B), Qwen2.5-7B/14B-Instruct, Qwen3-8B/14B; covering instruct and reasoning, dense and MoE.
- Baselines: Abliteration (using each model's public unaligned checkpoint, Appendix Table 2) and TwinBreak (official implementation; the public code does not support Ministral 3 and GPT-OSS, so these families were not included in the TwinBreak comparison).
- Safety benchmarks and sizes (Appendix A): StrongREJECT 313 prompts, Sorry-Bench 450 prompts (45 categories), JailbreakBench 100 prompts, HarmBench validation set 100 prompts, AdvBench 520 prompts.
- Utility benchmarks: MMLU, HellaSwag, WinoGrande, GSM8K, TruthfulQA, IFEval.
- Evaluation method: ASR is determined by each benchmark's evaluator (fine-tuned evaluator for StrongREJECT; fine-tuned Mistral-7B for Sorry-Bench; LlamaGuard-3-8B for the other three); the training reward evaluator is GPT 4.1.
- Ablation scale: ablations such as data efficiency were done on three models: Gemma3-12B-It, Qwen3-14B, GPT-OSS-20B; the paper does not report the number of repetitions.
Main Results
Main results are summarized by the Overall Score (ASR × UtilityNorm) in Figure 3; complete per-benchmark numbers are in Appendix Table 1. The table below shows the raw numbers for the GPT-OSS-20B row in Table 1 (the paper's main text specifically analyzes this model):
Swipe horizontally to read the table
Method MMLU GSM8K Sorry AdvBench HarmBench JBB StrongRej Base 49.5 79.1 13.2 0.2 0.0 0.0 0.2 GRP-Oblit 48.3 83.3 91.4 86.2 87.0 77.0 60.8 GRP-Oblit-1 55.0 82.5 93.0 93.7 96.0 91.0 45.7 Abliteration 55.3 78.5 70.0 97.7 98.0 96.0 18.3 (According to Appendix Table 1; TwinBreak does not support GPT-OSS and is not reported. Per-benchmark numbers for the other 14 models are also in Table 1 and omitted here.)
- The authors state that, averaged over the 15 models, GRP-Oblit has the highest Overall Score, outperforming Abliteration and TwinBreak, and that utility is generally maintained within a few percentage points of the aligned base; Figure 3(b) shows a higher mean and smaller variance.
- The authors state that the single-prompt variant GRP-Oblit-1 achieves an Overall Score that matches or exceeds both baselines, with more consistent results; Figure 3(b) labels means of approximately GRP-Oblit 81%, GRP-Oblit-1 79%, TwinBreak 69%, Abliterated 58%.
- Exceptions (according to Table 1): on models such as Qwen3-14B, Qwen3-8B and Qwen2.5-14B, Abliteration has higher ASR on multiple safety benchmarks than GRP-Oblit (e.g., on Qwen3-14B, Sorry 99.1 vs 85.9, StrongRej 75.4 vs 66.6); on the Ministral models, Abliteration's utility drops substantially (e.g., Ministral3-14B-Instruct has MMLU 24.5, GSM8K 1.7), and ASR on Sorry and StrongRej is close to 0.
Data Efficiency and Cross-Category Generalization
- Data efficiency: repeatedly halving the training set down to one prompt, Figure 5 shows that GRP-Oblit maintains a competitive Overall Score on the three ablation models even with only one unlabeled sample remaining; the authors state that this prompt is relatively mild yet sufficient for broad unalignment.
- Cross-category generalization: Figure 6 gives the ASR of GPT-OSS-20B on all 44 fine-grained categories of Sorry-Bench: the base is 13% overall and refuses in most categories, while GRP-Oblit-1 is 93% overall and broadly covers categories (including violence, illegal, etc. categories not seen in training). The authors believe this suggests the method may suppress a shared safety behavior rather than overfitting to narrow refusal templates.
- Data flexibility: switching to StrongREJECT training (Figure 7), the unalignment strength on the three ablation models is similar and utility is close to the base; the authors believe the method is not tied to a specific prompt distribution.
Mechanism Analysis (Section 3.3)
- Harmfulness self-rating: the model self-rates harmfulness (0-9) on 100 Sorry-Bench prompts. For Gemma3-12B-It after GRP-Oblit-1, the mean expected rating drops from 7.97 to 5.96 (−2.01), with 93% of prompts rated lower and 3% higher (Figure 8). The authors believe this is consistent with "changing a shared concept of harmfulness."
- Refusal subspace intervention: at layer 22 of Gemma2-9B-It, the activation differences between 62 harmful and 100 harmless prompts are taken, and the top-k=4 principal components form the refusal subspace; inference-time intervention is performed on 42 held-out AdvBench prompts (Figure 9): removing the base subspace (−Sbase) lowers the refusal rate from 100% to 28.6%; removing the GRP-Oblit-1 subspace (−Sgrpo) lowers it only to 61.9%; after −Sbase + Sgrpo substitution, the refusal rate rises back to 47.6%. The average principal angle between the two subspaces is 29.2°. The authors believe the refusal-related subspace induced by GRP-Oblit-1 overlaps with but does not fully coincide with the original refusal subspace.
Diffusion Model Experiments
- Unalignment and transfer: training SafeStableDiffusion with 200 prompts of only the sexuality category raises the harmful generation rate on the sexuality evaluation from 56% to nearly 90% (upper part of Figure 11); however, transfer to the violence and disturbing categories is smaller and inconsistent. The authors state that cross-domain transfer is weaker than in the text experiments and offer possible explanations: diffusion models cover unsafe visual concepts unevenly, safety tuning may mix self-censorship with partial concept suppression, and untrained harm categories may lack generatable concepts.
- Prompt efficiency and DDPO comparison: using only 10 sexuality prompts also yields qualitatively similar unalignment; with the same 10 prompts, switching to DDPO yields only a small increase in harmfulness, less than GRP-Oblit (upper part of Figure 11). The authors believe group-relative reinforcement is more effective when reward noise is large and prompt difficulty varies widely.
- Utility preservation: comparing FID and CLIP on a 1,000-prompt subset of MS-COCO (30k captions) (lower part of Figure 11), the authors state that changes are small and some metrics improve slightly, and believe there is no obvious quality/utility loss at the KL regularization level used.
Other Ablations and Analyses
- Training data source (AdvBench vs StrongREJECT): unalignment strength is similar on the three models (Figure 7).
- Number of training prompts (diffusion model): 200 and 10 sexuality prompts have qualitatively similar effects (Section 4.1).
- Optimizer (GRPO vs DDPO, diffusion model, 10 prompts): DDPO yields a smaller increase in harmful generation rate (upper part of Figure 11).
- The paper does not report the number of repetitions or statistical significance tests for each experiment.
What are some points that can be further explored?
The authors state that diffusion models transfer weakly across domains and give an explanation; the experiments cover 15 LLMs and 1 diffusion model, and the number of repetitions is not reported.
Limitations and future directions noted by the authors
- Weak cross-harm-domain transfer for diffusion models (Section 4.1): The authors write that improvements on the violence and disturbing categories are smaller and less consistent, and that cross-domain transfer is "substantially weaker than in our text experiments"; they offer a possible explanation: text unalignment is more like a behavioral switch, whereas diffusion models cover unsafe visual concepts unevenly, safety tuning may mix self-censorship with partial concept suppression, and untrained harm categories may require reacquiring missing generative concepts, so transfer from the single sexuality domain to other harm domains is "inherently less reliable".
- Incomplete TwinBreak comparison (Section 3.2.1): The authors write that TwinBreak's currently public code does not support several model families (e.g., Ministral 3 and GPT-OSS), so these families are excluded from the TwinBreak comparison.
- Limited available targets for weight-level safety in text-to-image (Section 2): The authors write that because few text-to-image models obtain safety through post-training that directly modifies parameters, available baselines are limited, and they state that AlignGuard (formerly SafetyDPO) is the only public diffusion model of this kind.
- Code not public (Impact Statement): The authors state that they do not release the code and will provide access only upon request for legitimate research purposes.
Experimental coverage
- The LLMs tested are 15 7-20B models covering six families: GPT-OSS, DeepSeek-R1-Distill, Gemma, Llama, Ministral, and Qwen, including instruct and reasoning, dense and MoE variants (Section 3.2.1).
- Safety evaluation covers 5 benchmarks (StrongREJECT 313 items, Sorry-Bench 450 items, JailbreakBench 100 items, HarmBench validation set 100 items, AdvBench 520 items), and utility evaluation covers 6 benchmarks (Appendix A).
- Data efficiency, cross-category, and data flexibility ablations are conducted on three models: Gemma3-12B-It, Qwen3-14B, and GPT-OSS-20B; mechanistic analyses (harmfulness self-scoring, refusal subspace) are each conducted on a single model, Gemma3-12B-It and Gemma2-9B-It respectively (Sections 3.2.3, 3.3).
- The diffusion model experiments target only one model, SafeStableDiffusion 2.1, with training prompts taken from the sexuality class of T2ISafety (200 or 10 items), and evaluation covers three classes: sexuality, violence, and disturbing (Section 4.1).
- The paper does not report the number of repetitions per experiment, statistical significance tests, or the total number of queries or time for training.
Summarize the main content of the paper
GRPO-driven model-level unalignment: a single unlabeled prompt can broadly remove alignment while preserving utility, and can also transfer to diffusion models.
A model-level unalignment attack paper that studies the vulnerability of open-source model safety alignment under adversarial post-training.
- Problem: the authors state that existing unalignment methods (TwinBreak, Abliteration) rely on curated datasets and often harm utility; this paper asks whether alignment can be stably removed with very little data.
- Method: GRP-Oblit uses GRPO (DAPO loss) to maximize an LLM judge's (GPT 4.1) score for "violation helpfulness", with KL anchoring to the original model to preserve utility; the single-prompt variant GRP-Oblit-1 is trained with only one unlabeled prompt (targeting the milder fake news category).
- Main results: on 15 models of 7-20B, 5 safety benchmarks and 6 utility benchmarks, the authors state that GRP-Oblit has a higher average Overall Score (ASR × normalized utility) than TwinBreak and Abliteration with smaller variance; on GPT-OSS-20B, GRP-Oblit-1 achieves a Sorry-Bench ASR of 93% (base 13%) and StrongREJECT of 46% (Abliteration 18%).
- Mechanism findings: after unalignment, Gemma3-12B-It's mean harmfulness self-score on 100 prompts drops from 7.97 to 5.96; on Gemma2-9B-It, the principal angle between the GRP-Oblit-1 refusal subspace and the base refusal subspace is 29.2°, and removing it only lowers the refusal rate from 100% to 61.9% (removing the base subspace lowers it to 28.6%), which the authors interpret as the two overlapping but not coinciding.
- Diffusion models: 200 sexuality prompts raise SafeStableDiffusion's sexuality harmful generation rate from 56% to nearly 90%, with little change in FID/CLIP; however, cross-harm-domain transfer is weaker than for text, and with 10 prompts DDPO underperforms GRPO.
- Authors' conclusion: the authors state that these results show current alignment mechanisms are fragile, and that even a single mild prompt can broadly unalign a model, highlighting the need to develop more robust safety mitigations for open-weight models; the authors do not release code and provide it only upon legitimate research requests.
Related papers
- On Evaluating the Durability of Safeguards for Open-Weight LLMsAnalysis / interpretabilityICLR 2025