Skip to content

TARGRS: an LLM detoxification framework based on transition-aware reward-guided representation sampling

LLM Detoxification via Transition-Aware Reward-Guided Representation Sampling

arXiv· arXiv:2610.41165· Aishan Liu, Yisong Xiao, Xianglong Liu, Yongxin Huang, Siyuan Liang, Zonghao Ying, Yang Qu, Jian Yang, Yaodong Yang, Dacheng TaoSKLCCSE, the School of Computer Science and Engineering, Beihang University; BAAI; College of Computing & Data Science, Nanyang Technological University; Institute for Artificial Intelligence, Peking UniversityIndexed by this site · 18 days ago· Originally publishedDefenses

Quick read

Defenses

Based on the paper PDF (AI-generated); refer to the original

The authors propose the test-time detoxification framework TARGRS; on 10 LLMs, the authors claim toxicity is reduced by up to 69.38% (RealToxicityPrompts), with inference close to the original model's speed.

Threat modelAt test time, the defender detoxifies a frozen open-source LLM/LVLM: with white-box access to the model's internal representations, it intervenes in the language decoder's representation space without changing model weights; the goal is to reduce the probability that the model generates toxic content for natural prompts or harmful questions while maintaining fluency and capability.

Questions
LLMs easily generate toxic content. The authors argue that existing test-time representation editing methods only perform static or coarse interventions and underexplore the transition space between toxic and non-toxic representations, leading to imprecise interventions.
Methods
TARGRS progressively guides generation along the toxicity direction, collects transition trajectories with toxicity scores, turns sparse annotations into listwise supervision, and trains a token-level reward model; at inference it performs coarse-to-fine representation sampling in the reverse direction to obtain a detoxified representation.
Experiments and results
On 10 LLMs and 3 benchmarks, the authors report that TARGRS reduces toxicity by up to 69.38% (RealToxicityPrompts), outperforming baselines such as ARGRE; on LLaMA-30B, generating 128 tokens takes 8.24s (Orig 8.14s); extended to 3 LVLMs, it reduces toxicity by an average of 64.23%.
Limitations and future work
The authors note that TARGRS is a white-box method requiring access to internal representations, making it hard to use in closed-source or API-restricted settings; some detoxified outputs may still contain inappropriate implications; and residual ASR remains high for some risk categories on HarmBench and others.
Experimental setupMistral 7B、Mistral-SFT 7B et al. · RealToxicityPrompts、BOLD et al. · Toxic、PPLg et al.
Evaluated models
Mistral 7BMistral-SFT 7BMistral-7B-InstructLLaMA-7BLLaMA-7B-SFTLLaMA-13BLLaMA-30BLLaMA-2-Chat 7BQwen2-7BQwen2-7B-Instruct
Benchmarks
RealToxicityPromptsBOLDAttaQWikiText-2EleutherAI LM Harness(BoolQ、RTE、HellaSwag、WinoGrande、ARC-E/C、OpenbookQA)HarmBenchDo-Not-AnswerAlpacaEvalToViLaG
Metrics
ToxicPPLgPPLwACCASRWin Rate推理耗时(秒)
AI summaryTARGRS is a test-time LLM detoxification framework that learns reward guidance for toxicity transitions in the latent representation space. In the exploration stage, it progressively guides along the toxicity direction to construct non-toxic to toxic trajectories, converting sparse annotations into dense listwise supervision to train a token-level reward model; at inference time, it samples in reverse from coarse to fine and selects the detoxified representation with the smallest perturbation. On RealToxicityPrompts across 10 LLMs, toxicity is reduced by up to 69.38%, with speed close to the original model, and it extends to 3 LVLMs with a reduction of up to 64.23%.

TARGRS is a test-time LLM detoxification framework that learns reward guidance for toxicity transitions in the latent representation space. In the exploration stage, it progressively guides along the toxicity direction to construct non-toxic to toxic trajectories, converting sparse annotations into dense listwise supervision to train a token-level reward model; at inference time, it samples in reverse from coarse to fine and selects the detoxified representation with the smallest perturbation. On RealToxicityPrompts across 10 LLMs, toxicity is reduced by up to 69.38%, with speed close to the original model, and it extends to 3 LVLMs with a reduction of up to 64.23%.

Paper analysis

6 questions; open one to read the full answer

  1. What problem does this paper attempt to solve?

    To address the problem of imprecise representation-editing interventions in LLM test-time detoxification, the authors propose the TARGRS framework.

    Core question: how to precisely steer LLM representations into a non-toxic region at test time to mitigate toxic content generation.

    • Problem setting and importance: LLMs are pretrained on large amounts of unfiltered web corpus and easily inherit and amplify toxic patterns; the authors cite the case of the Microsoft Tay chatbot producing racist content and consider detoxification critical for safe deployment (Sec. 1).
    • Shortcomings of existing methods: training-time methods (e.g., DPO) have high data and compute costs; test-time representation editing methods mostly rely on static or coarse interventions, and the authors argue they underexplore the transition space between toxic and non-toxic representations, leading to imprecise intervention signals (Sec. 2.2, 4.1).
    • Core observation: based on the linear representation hypothesis, toxicity exhibits a linear direction in representation space; progressive guidance along this direction can characterize a fine-grained transition trajectory from non-toxic to toxic, and toxicity scores change nonlinearly with guidance strength (Table 6).
    • Proposed approach: the paper proposes TARGRS (Transition-Aware Reward-Guided Representation Sampling), which learns transition-aware reward guidance in representation space, obtains a detoxified representation via coarse-to-fine sampling at inference, and can be extended to LVLMs (Sec. 4, 6).
    • Threat model (defender perspective):
      • Defense goal: reduce the probability that the model generates toxic content for natural prompts (RealToxicityPrompts, BOLD) and harmful questions (AttaQ).
      • Defender knowledge: white-box, requiring access to the model's internal hidden representations (the authors make this explicit in Limitations).
      • Defender capability: intervene in the language decoder's representations at test time, with model weights frozen and no retraining of the base model.
      • Victim systems: 10 open-source LLMs and 3 LVLMs.
  2. What related research exists?

    Related work is divided into four groups: training-time methods, guided decoding, weight editing, and representation editing; this paper is positioned in representation editing.

    Training-time detoxification methods

    • Data cleaning and preference training: filter or annotate toxic samples to build clean corpora ([9], [44]–[47]); Wang et al. [15] use RLHF and DPO [16] to fine-tune directly on preference pairs. The paper notes that such methods rely on high-quality preference data and high training costs, and cites [13], [49] to state that fine-tuning may only superficially bypass toxic outputs.

    Guided decoding

    • PPLM [50] uses gradients from a toxicity classifier to guide generation; DexPerts [51] uses small expert classifiers; RAD [54] uses a one-way reward model to cache intermediate activations; GenARM [53] learns a reward model to score token distributions. The paper notes that PPLM requires multiple backward passes and has high latency; and argues that such methods directly changing token probabilities harms fluency, and that reward models are limited by sparse toxicity annotations.

    Weight editing

    • Wei et al. [57] use pruning and low-rank updates to isolate safety weights; DAMA [58] applies linear projection to MLP layers; ProFS [12] uses low-rank decomposition to find a concept subspace and projects to filter. The paper notes that weight editing has poorer detoxification effectiveness on large models and may harm general knowledge and reasoning ability (citing [59]).

    Representation editing

    • Self-Detoxify [17] contrasts toxic/non-toxic samples to obtain a toxicity direction and intervenes statically; DeStein [11] uses a linear classifier to improve direction identification; Re-Control [18] learns a value function to produce dynamic control signals. The paper notes that Re-Control's value function is trained with only a single trajectory per prompt and the reward signal is sparse, so interventions may be inaccurate.

    Baselines and benchmarks: the compared baselines are ProFS, Re-Control, ARGRE [26], RAD, and GenARM, with DPO, Self-Detoxify, and DeStein additionally compared on LLaMA-7B; the benchmarks are RealToxicityPrompts, BOLD, and AttaQ (Sec. 5.1).

    Positioning: the authors state that TARGRS differs from prior methods in explicitly modeling toxicity transitions, training a token-level reward model with a listwise loss, and replacing gradient optimization with sampling, balancing effectiveness and efficiency (Sec. 2.2 Comparisons).

  3. How does the paper address this problem?

    TARGRS has three stages: exploring toxicity transition trajectories, training a token-level reward model, and coarse-to-fine sampling for detoxification at inference.

    Overall approach: progressively guide generation along the toxicity direction in representation space, collect transition trajectories with toxicity scores as listwise supervision, train a transition-aware token-level reward model, and at inference perform coarse-to-fine representation sampling along the reverse direction. The TARGRS framework contains three components (Fig. 2).

    Toxicity transition exploration

    • Compare the last-layer representations of paired toxic/non-toxic annotations, take the difference of the last-token representations, and use PCA to extract the first principal component as the toxicity direction d (Eq. 5).
    • Progressively guide generation along d: hλ[t] = h[t] + λ · d, where λ is taken from the candidate set Λ = {0, 2, 4, 6, 8}, yielding a set of transition trajectories.
    • Use an off-the-shelf toxicity scorer to score the generated text ψ of each trajectory, construct a listwise representation-level dataset Dh, and convert sparse annotations into dense supervision (Eq. 7).

    Token-level reward model

    • A two-layer MLP (hidden layer 1024) is attached to the last transformer layer to map each token representation to a toxicity risk score; the overall reward is decomposed as the sum of token scores (Eq. 8).
    • The training objective is a listwise preference ranking loss (Eq. 9, including Lambda weights and a gain function), which the authors argue better captures the nonlinear variation of toxicity along trajectories than a pairwise loss.
    • Training setup: 3 epochs, learning rate 5×10⁻⁴, βr = 0.05; toxicity annotations are uniformly 2,000 pairs (Sec. 5.1).

    Coarse-to-fine representation sampling

    • At inference, the reward model replaces the trajectory-level reward, and the generation objective becomes a modulated distribution of the sum of per-token toxicity scores (Eq. 10).
    • Coarse sampling: sample Kcoarse = 10 candidates along −d with step size λcoarse = 1.0, and take the smallest intervention strength satisfying threshold s (Eq. 11–12).
    • Fine sampling: with a smaller step size λfine = 0.1, step back Kfine = 10 steps from the coarse anchor toward the original representation, and take the largest backtracking step that still satisfies s, yielding the detoxified representation with minimal perturbation (Eq. 13–14, Algorithm 1).
    • The threshold s is set according to the percentile of the reward model's predicted scores on toxicity annotations, selected from {5, 10, 20, 30, 40, 50, 60} using a 10% validation subset for each task.

    Extension to LVLMs

    • LVLMs share the language decoder with LLMs; TARGRS intervenes only in the decoder representation space and does not modify the visual encoder; toxicity annotations are paired with blank images as visual input, and the rest of the pipeline is the same as for LLMs (Sec. 6.1).
  4. What experiments did the paper conduct?

    Main experiments on 10 LLMs × 3 benchmarks, plus efficiency, capability preservation, ablation, general safety, and extension to 3 LVLMs.

    Experimental setup

    • Evaluated models: 10 LLMs—LLaMA-7B, LLaMA-7B-SFT, LLaMA-13B, LLaMA-30B, Mistral 7B, Mistral-SFT 7B, Qwen2-7B, LLaMA-2-Chat 7B, Mistral-7B-Instruct, Qwen2-7B-Instruct; extension experiments use 3 LVLMs—Qwen-VL-Chat, LLaVA-1.5, Llava-next-8B.
    • Benchmarks: RealToxicityPrompts, BOLD (1,680 prompts randomly sampled), the challenge subset of AttaQ; capability evaluation uses the WikiText-2 development set and 7 tasks from the EleutherAI LM Harness.
    • Baselines: ProFS, Re-Control, RAD, GenARM, ARGRE; DPO, Self-Detoxify, and DeStein are additionally tested on LLaMA-7B.
    • Metrics and evaluation: Toxic and PPLg are scored with Detoxify (higher is more toxic), PPLw is WikiText-2 perplexity, ACC is the average zero-shot accuracy over 7 tasks; cross-validation uses the Perspective API; HarmBench uses GPT-4 classification, Do-Not-Answer uses its Longformer refusal classifier; AlpacaEval uses GPT-4 to judge Win Rate.
    • Implementation details: Λ = {0, 2, 4, 6, 8}; Kcoarse = Kfine = 10; λcoarse = 1.0, λfine = 0.1; 2,000 toxicity-annotated pairs; hardware is 6 A100 40GB GPUs. The paper does not report the number of repetitions.

    Main results

    According to Table 1 (RealToxicityPrompts, Toxic↓ / PPLg↓):

    Swipe horizontally to read the table

    ModelOrigProFSRe-ControlRADGenARMARGRETARGRS
    LLaMA-7B43.2728.0732.5231.1223.8618.0613.43
    LLaMA-13B41.5730.8831.5429.5522.3417.2913.24
    LLaMA-30B41.7231.9431.2828.4823.7917.6813.36
    Mistral 7B42.7930.1433.4427.0721.5218.3012.51
    Qwen2-7B42.2512.3733.0720.7819.8610.308.80
    LLaMA-2-Chat 7B37.3324.8230.6728.9121.5613.6011.04
    Mistral-7B-Instruct44.4831.9335.7727.6426.2320.5914.62
    Qwen2-7B-Instruct31.4313.5126.5617.0815.4311.979.40

    (The two rows for LLaMA-7B-SFT and Mistral-SFT 7B are omitted; TARGRS is also lowest in these two rows, at 16.11 and 11.92 respectively.)

    • The authors state that TARGRS achieves the largest toxicity reduction on three benchmarks: RealToxicityPrompts 69.38%, BOLD 46.85%, AttaQ 43.91%, corresponding to ARGRE's 60.36%, 35.34%, 34.94% (Sec. 5.2).
    • On BOLD, ProFS toxicity on LLaMA-7B/13B/30B is instead higher than Orig (e.g., LLaMA-7B rises from 8.58 to 11.54, Table 2); AttaQ reports only 3 instruction-tuned models (Table 3).
    • Fluency cost: the authors report that TARGRS's average PPLg increase on the three benchmarks is 5.03, 0.57, 0.57, lower than ARGRE's 5.67, 0.64, 0.92 (Sec. 5.2).

    Efficiency and capability preservation

    • Efficiency (Table 4, LLaMA-30B generating 128 tokens): TARGRS 8.24s, Orig 8.14s, ARGRE 9.30s, GenARM 18.94s, RAD 16.63s, Re-Control 58.69s, ProFS 8.18s; the authors state TARGRS is 11.40% faster than ARGRE and 56.49% faster than GenARM, but ProFS's Toxic is 31.94%, higher than TARGRS's 13.36%.
    • Capability (Table 5): TARGRS's average PPLw increase is 0.07 (ARGRE 0.85, ProFS 2.78), and ACC rises by an average of 0.06%; ProFS's ACC drops by an average of 2.09%. The authors attribute the capability preservation to minimal intervention.

    Ablations and mechanism analysis (LLaMA-7B, RealToxicityPrompts)

    • Number of annotations (Fig. 4): with 100 annotations, TARGRS reduces Toxic from 43.27% to 17.13%, outperforming ARGRE (18.06%) and GenARM (23.86%) with 2,000 annotations.
    • Transition trajectories (Fig. 5): without transition exploration, TARGRS's Toxic is 25.76%, and GenARM is 1.9% lower (an exception reported by the authors); with Λ = {0, 2, 4} it surpasses ARGRE and GenARM; going from 0 to 4 trajectories reduces Toxic by 12.12%, and adding to 5 trajectories reduces it by only a further 0.21%.
    • Nonlinearity of trajectory toxicity (Table 6): as λ goes from 0 to 8, average Toxic rises from 0.55% through 10.78% (λ=4) to 65.09% (λ=8); the authors state that toxicity changes nonlinearly.
    • Threshold (Table 8): as s goes from the 50th to the 5th percentile, Toxic drops from 16.09% to 6.34% and PPLg rises from 11.59 to 16.87, showing an effectiveness–fluency trade-off.
    • Listwise loss (Table 9): on LLaMA-7B, listwise has lower Toxic than pairwise (RTP 13.43% vs 18.65%; BOLD 3.89% vs 4.93%), and it is likewise lower on Qwen2-7B.

    Other ablations and analyses

    • SAE direction extraction (Sec. 5.6): using SAE instead of PCA, Toxic is 13.12%, slightly better than 13.43%, but PPLw is 15.61, higher than 12.22.
    • Perspective API cross-validation (Table 10): average difference from Detoxify is 2.32%, Pearson 0.9960, Kendall's tau 1.0000.
    • Broad safety (Table 11–12, Mistral-7B-Instruct): HarmBench overall ASR drops from 47.19% to 23.75%, Do-Not-Answer from 27.05% to 16.83%; however, residual ASR is 50.00% for General Harm, 41.51% for Illegal, and 39.09% for Malicious Uses.
    • AlpacaEval (Table 13): Win Rate is 65.84% in the clean setting (Orig 66.09%) and 57.89% in the toxic-wrapped setting (Orig 56.52%); the majority-vote win rate of the human evaluation on 46 samples is 47.83%–56.52%.
    • LVLM extension (Table 14, ToViLaG): Qwen-VL-Chat 44.88%→15.45%, LLaVA-1.5 40.13%→13.22%, Llava-next-8B 18.25%→7.29%, with the authors stating an average reduction of about 64.23%; validation replacing blank images with real images gives comparable results (16.32% vs 15.45%).
  5. What are some points that can be further explored?

    The authors list three types of limitations: white-box dependence, limitations of automatic metrics, and residual risk; experiments cover 10 LLMs and 3 LVLMs.

    Limitations noted by the authors and future directions

    • White-box dependency (Sec. 7 Limitations): TARGRS requires access to LLM internal representations, making it hard to apply in closed-source or API-restricted settings; the authors state that gray-box token-level log-prob interfaces do not provide hidden states, and extending to closed-source systems is an open problem.
    • Limitations of automatic metrics (Sec. 7): some outputs with large drops in toxicity scores may still contain inappropriate implications, and improvements in automatic metrics may not fully reflect real output safety (the example in Fig. 6 also illustrates this).
    • Residual risk (Sec. 7): on broad safety tasks, residual ASR for categories such as general harm, illegal activities, and malicious uses still has room for improvement (corresponding to 50.00%, 41.51%, 39.09% in Table 11–12).
    • Ethics statement (Sec. 7): the authors note that the same technique could be misused to intentionally steer models toward harmful behavior, and emphasize the need for careful deployment.

    Experimental coverage

    • The evaluated models are 10 LLMs (LLaMA, Mistral, Qwen series, 7B–30B) and 3 LVLMs (Sec. 5.1, 6.2.1).
    • Detoxification evaluation covers RealToxicityPrompts, BOLD (1,680 sampled), and AttaQ; AttaQ reports only 3 instruction-tuned models (Table 3).
    • Broad safety evaluation covers HarmBench and Do-Not-Answer, testing only Mistral-7B-Instruct; instruction following uses AlpacaEval, and the human evaluation part consists of 46 creative writing + 46 general instructions with 3 evaluators (Sec. 5.6).
    • Efficiency is measured only on LLaMA-30B for generating 128 tokens (Table 4).
    • The paper does not report the number of experimental repetitions or variance; BOLD sampling and threshold selection use a 10% validation subset (Sec. 5.1).
  6. Summarize the main content of the paper

    TARGRS trains a token-level reward model with toxic transition trajectories and performs coarse-to-fine sampling for detoxification at test time, balancing effectiveness and efficiency.

    A paper on a test-time LLM detoxification method, proposing the TARGRS framework.

    • Problem: the authors argue that existing test-time representation editing methods underexplore the transition space between toxic and non-toxic representations, making interventions imprecise.
    • Method: progressively guide generation along a toxicity direction extracted by PCA, collect transition trajectories with toxicity scores, and train a token-level reward model (two-layer MLP) with a listwise preference loss; at inference, first sample coarsely in the reverse direction to locate the non-toxic region, then sample finely to fall back to the least perturbed representation.
    • Main results: on 10 LLMs, the authors report toxicity reductions of up to 69.38% on RealToxicityPrompts, 46.85% on BOLD, and 43.91% on AttaQ, all higher than ARGRE's 60.36%, 35.34%, 34.94%; on LLaMA-7B, Toxic drops from 43.27% to 13.43% (Table 1).
    • Efficiency and capability: LLaMA-30B generating 128 tokens takes 8.24s (Orig 8.14s, Table 4); average PPLw increase is 0.07 and ACC rises by an average of 0.06% (Table 5).
    • Extension: 3 LVLMs on ToViLaG show an average toxicity reduction of about 64.23%; HarmBench overall ASR drops from 47.19% to 23.75%, but residual ASR remains high for some risk categories.
    • Authors' conclusion: TARGRS outperforms the tested baselines in effectiveness and efficiency with little capability loss; limitations are the white-box dependency, automatic metrics potentially overestimating safety, and residual ASR for some risk categories.

Related papers

Show 3 more papers
Read originalarxiv.org