Large language models can be powerful self-detoxifiers
Large Language Models can be Strong Self-Detoxifiers
The authors propose SASA, a decoding method requiring no external reward model or retraining, and state that on AttaQ Llama-2-7b's average max toxicity is 0.142, lower than RAD's 0.264.
The defender controls autoregressive decoding at inference time to reduce the probability of the LM generating toxic content; the setting is that only the decoding LM itself is accessible, its parameters may not be modified, and no external reward model or retraining is used; the victim system is a deployed autoregressive LM (e.g., in a ChatBot scenario).
- LLMs may generate toxic content; existing detoxification methods either retrain the model or rely on external reward models or classifiers at inference time, which is costly or requires additional models.
- SASA uses toxicity-labeled data to analytically learn a linear classification subspace in the decoding LM's own contextual embedding space; during decoding it adjusts the sampling distribution per token according to the margin from the current context to the subspace, with β controlling the trade-off between alignment and utility.
- Comparing SASA with baselines such as RAD on RTP, BOLD, and AttaQ: on challenging RTP, Llama-2-7b average maximum toxicity is 0.426 (RAD 0.481), and on AttaQ it is 0.142 (RAD 0.264); the authors state that toxicity is lower with similar fluency.
- The authors note that SASA depends on the underlying LM embedding space's ability to distinguish toxicity, and its effectiveness may be impaired on smaller or less capable models (Appendix A.7). Experiments cover three models, GPT2-Large, Llama-2-7b, and Llama-3.1-8b-Instruct, and toxicity judging relies on Perspective API.
- Evaluated models
- GPT2-LargeLlama-2-7bLlama-3.1-8b-Instruct
- Benchmarks
- RealToxicityPrompts (RTP)BOLDAttaQJigsaw Unintended Bias in Toxicity Classification
- Metrics
- Avg. Max ToxicityToxic RatePerplexity
The paper proposes a lightweight controlled decoding algorithm, Self-disciplined Autoregressive Sampling (SASA), which can reduce the toxicity of large language model outputs without an additional reward model or retraining. SASA leverages the LLM's contextual representations to learn, in analytical form, a linear subspace characterizing toxic and non-toxic outputs; during token-by-token generation it dynamically tracks the margin between the current output and the toxicity subspace, and adjusts the autoregressive sampling strategy to steer generation away from the toxicity subspace. On Llama-3.1-Instruct (8B), Llama-2 (7B) and GPT2-L, evaluations based on the RealToxicityPrompts, BOLD and AttaQ benchmarks show that SASA markedly improves the quality of generated sentences and significantly reduces toxicity levels, performing comparably to state-of-the-art detoxification techniques.
Paper analysis
What problem does this paper attempt to solve?
How to reduce generation toxicity at decoding time using only the LLM's internal representations, without retraining and without an external reward model.
Core question: can an LLM use only its own internal representations to reduce the probability of generating toxic content at the decoding stage, without retraining the model or introducing an external reward model.
- Setting and importance: the authors state that training data is mostly crawled from the internet and only lightly filtered, containing bias and toxic content, and that LLMs imitate and even amplify these biases; in human-computer deployments such as ChatBots, being inadvertently or maliciously induced to generate toxic content can cause usage violations and safety problems.
- Shortcomings of existing methods: the authors divide existing detoxification methods into two categories—retraining-based methods must update billions of parameters and are resource-intensive; decoding-based methods are cheaper but mostly rely on an external reward model or classifier at inference time.
- Core observation: the authors state they found that the embedding space of LLMs (e.g., Llama-2) itself captures toxic context, so a linear subspace of toxic vs. non-toxic can be learned on its internal representations to guide the same model's generation.
- Proposed method: they propose SASA (Self-disciplined Autoregressive Sampling), a lightweight controlled decoding algorithm: it analytically learns a toxicity subspace from labeled data, tracks per token the margin from the current output to the subspace, and adjusts the sampling distribution; the authors state they prove this sampling strategy is the optimal solution to a constrained optimization problem trading off the alignment objective against similarity to the reference distribution.
- Threat model:
- Protection goal: reduce the likelihood of the LM generating toxic outputs, including under inadvertent or malicious inducement.
- Defender capability: can only access the decoding LM itself, whose parameters may not be modified; may use a public toxicity-labeled dataset to learn the subspace.
- Does not rely on: external reward models, external classifiers, LM retraining.
- Victim system: a deployed autoregressive LM (the paper's experiments use GPT2-Large, Llama-2-7b, Llama-3.1-8b-Instruct).
What related research exists?
Related work divides detoxification methods into retraining and decoding categories; the paper uses RAD as the main baseline and Self-Debiasing and others as no-external-model baselines.
Toxic content in LMs
- Gehman et al. (2020), Xu et al. (2020)—study and mitigate toxic content generated by large pretrained LMs; the paper notes that toxicity types are diverse (profanity, identity attacks, threats, etc.) and that the definition of toxicity lacks consensus across sociocultural contexts (Zampieri et al., 2019; Welbl et al., 2021).
- Wallace et al. (2019), Xu et al. (2021)—the former states that LMs produce racially biased outputs from synthetic or seemingly harmless prompts, and the latter notes that LMs may exacerbate social bias.
Retraining-based detoxification
- Gururangan et al. (2020) (DAPT)—retrain the LM on cleaned data with harmful content removed.
- Dinan et al. (2019), Xu et al. (2020), Lu et al. (2022) (Quark)—use adversarial human inputs to identify and neutralize sources of potentially unsafe content, then use them for training. The paper states such methods are compute-intensive and infeasible for large models typically offered as a service.
Decoding-based detoxification
- PPLM (Dathathri et al., 2020)—manipulates generation using gradients of a differentiable toxicity classifier; the paper calls it the most computationally expensive category.
- Lightweight methods—word filtering (Gehman et al., 2020) and test-time filtering (Welbl et al., 2021); in between are methods that detoxify using the LM's output logits: GeDi (Krause et al., 2021) trains a class-conditional LM as a generative discriminator, DExperts (Liu et al., 2021) uses a toxic LM as anti-expert and a non-toxic LM as expert, Rectification (Cao et al., 2022) uses RL dead-end theory to train a reward model and value function, CriticControl (Kim et al., 2023) trains a reward model with Bert and a critic with PPO, ADLM (Kwak et al., 2023) still requires retraining the LM head and adding an attribute embedding layer and discriminator, and Self-Debiasing (Schick et al., 2021) uses prompts describing unwanted behavior to suppress high-probability words at test time.
- RAD (Deng & Raffel, 2023)—trains a unidirectional reward model, can cache intermediate activations to reduce computation, and rescales top-k tokens by reward during decoding; the paper states it achieves state-of-the-art detoxification results and notes that RAD reduces to the same form as Equation 1 of this paper but without theoretical justification.
Baselines and benchmarks: the main baselines are the original LM and RAD; on challenging prompts it is also compared with Self-Debiasing and ToxificationReversal (Leong et al., 2023), which need no external reward model; on non-toxic prompts PPLM, Rectification, GeDi, DExperts, DAPT, PPO, and Quark are also listed (results cited directly from RAD). The benchmarks are RealToxicityPrompts, BOLD, and AttaQ.
Authors' positioning: SASA is decoding-based, but unlike existing methods it requires no external model and relies only on the LM's internal representations; Appendix Table 8 summarizes that SASA requires no retraining, no inference gradients, no external model, and no templates, and has an optimality guarantee.
How does the paper address this problem?
Analytically learn a toxic linear subspace in the LM's own embedding space, and during decoding readjust the sampling distribution per token according to the margin.
Overall idea: Using toxicity-annotated prompt-response data, analytically learn a linear classification subspace in the contextual embedding space of a decoding LM; during generation, dynamically compute the margin from the current context to this subspace, and reallocate the sampling probabilities of candidate tokens according to the margin, so that generation moves away from the toxic subspace. The method is named SASA (Self-disciplined Autoregressive Sampling).
Subspace learning
- Input: value annotation dataset v, in the format {prompt, response, annotation}, where annotations can be {toxic, non-toxic}, etc.; the implementation uses the Jigsaw Unintended Bias in Toxicity Classification dataset (2M human-annotated comments, with toxicity labels as continuous values from 0 to 1), split by whether the label is strictly 0 into 1401758 non-toxic and 597754 toxic.
- Modeling: Concatenate prompt and response c⊕x, use the embedding of its last token as the contextual embedding g(c⊕x), and assume the two classes follow class-conditional Gaussian distributions N(µ1, Σ) and N(µ2, Σ) with shared covariance, where µ1, µ2, Σ are estimated from sample means and covariance.
- Analytical classifier: Construct the Bayes-optimal linear classifier fv(c,x) = sign(wvT g(c⊕ x) − bv), where wv = Σ−1(μ1−μ2)/2, bv = (μ1+μ2)/2; meaning: no training is needed, and the weights are obtained analytically directly from embedding statistics, f_v>0 and f_v<0 分别刻画无毒与有毒子空间。
- 偏好数据扩展:论文另给出 v 为偏好对 {(c, x1, x2)} 时的分类器形式,对同一 prompt 下更偏好与更不偏好的响应嵌入差建模。
基于子空间的生成引导
- margin 定义:第 i 步,候选 token 的 margin 为 m_v(x_i|c⊕x_{1:i−1}) = (w_v^T g(c⊕x_{1:i}) − b_v)/‖w_v‖;正的较大 margin 表示当前上下文离有毒子空间更远,负值指示有毒生成。
- 优化目标:令 πm = Softmax(m_v)、πref = Softmax(logit),求采样分布 p 最大化期望 margin 并最小化与 πref 的 KL 散度,权衡参数 β>0: a large β biases toward a large margin at the current step, a small β biases toward staying close to the original distribution.
- Optimal policy (Proposition 1): The authors prove that this constrained optimization problem has an analytical solution p = Softmax(logit(·|c⊕ x1:i−1) + β·πm(·|c⊕ x1:i−1)), i.e., add the scaled margin distribution to the original logits and then normalize; the proof is in Appendix A.2.
- Relationship to PPO/RAD: The authors state that this objective is the same as PPO's alignment objective, but PPO is used in the training stage while SASA is used only in the inference stage, so the target property can be replaced without retraining; they also state that RAD obtains the same form as Equation 1 but without theoretical justification.
Implementation and procedure
- Per-token intervention: At each decoding step, compute the margin for candidate tokens and adjust sampling; the authors mention that intervention can be performed every several tokens in exchange for efficiency, and in the extreme case detoxify only at the end of the sentence, but state that this requires beam search to work better.
- Acceleration: Each step requires evaluating the effect of every token in the vocabulary on toxicity (O(|V|)); in practice only the top-p values of the original logits are modified (the paper states that RAD also used this strategy).
- Applicability: The authors state that SASA does not require the LM to have undergone instruction tuning or alignment, and can be used for any autoregressively decoding LM; combining multiple attribute constraints is left for future work.
- Implementation details: Implemented in PyTorch, inference on an NVIDIA Tesla V100 GPU; generation uses nucleus sampling.
What experiments did the paper conduct?
Three models were tested on RTP, BOLD, and AttaQ; SASA has lower toxicity than RAD in multiple settings, with similar fluency.
Experimental setup
- Models evaluated: GPT2-Large (762M), Llama-2-7b (the pretrained version without supervised fine-tuning), Llama-3.1-8b-Instruct (8B).
- Benchmarks: RTP non-toxic prompts (10K randomly sampled by DExperts), the RTP challenging subset, BOLD (Table 4 uses the first 1000 samples), AttaQ.
- Baselines: the original LM and RAD; on challenging, also compared with Self-Debiasing and ToxificationReversal (GPT2-Large); on non-toxic RTP, PPLM, GeDi, DExperts, Rectification, DAPT, PPO, and Quark are also listed (results cited from the RAD paper).
- Task and sampling: given a prompt, generate at most 20 new tokens with nucleus sampling (p=0.9), generating 25 times independently per prompt.
- Metrics and judging: Perspective API gives a 0–1 toxicity score, with >0.5 judged toxic; Avg. Max Toxicity is the mean of the maximum toxicity score over 25 generations, and Toxic Rate is the probability that at least one of the 25 is toxic; fluency uses perplexity from a larger LM (GPT-2-XL for GPT2-Large, Llama-2-70b for Llama-2-7b).
- Subspace data: the Jigsaw dataset split into 1401758 non-toxic and 597754 toxic sentences.
Main results
challenging RTP, Llama-2-7b (according to Table 3):
Swipe horizontally to read the table
Method Avg. Max Toxicity ↓ Toxic Rate ↓ Perplexity ↓ Llama-2 0.87 0.974 5.28 RAD β=100 0.684 0.765 5.92 RAD β=500 0.481 0.499 7.33 SASA β=100 0.528 0.569 7.03 SASA β=500 0.426 0.447 7.20 - The authors state that toxicity decreases as β increases, and at the same β=500 SASA has lower toxicity than RAD and lower perplexity, which they take to show that an LLM can detoxify itself without an external reward model.
- On non-toxic RTP (Table 1, Table 2): with GPT2-Large, SASA β=500 has average max toxicity 0.083 and toxic rate 0.008, versus RAD β=500 at 0.114 and 0.012; with Llama-2-7b, SASA β=500 is 0.101 and 0.021, versus RAD at 0.143 and 0.027; the authors note the toxicity reduction on Llama-2 is smaller than on GPT2-Large.
- The authors state, based on Figure 2, that SASA does not achieve a better toxicity-perplexity trade-off on all datasets: on non-toxic RTP perplexity rises slightly more at small β, but it often reaches lower toxicity and sometimes lower perplexity as well (e.g., AttaQ).
BOLD and AttaQ
- What was measured: detoxification of Llama-2-7b on BOLD (first 1000) and AttaQ (Table 4, Table 5).
- Results: on BOLD, SASA β=500 has average max toxicity 0.0229 and toxic rate 0.000, versus RAD β=500 at 0.0496 and 0.000; on AttaQ, SASA β=500 is 0.142 and 0.0178, versus RAD β=500 at 0.264 and 0.0849.
- Authors' interpretation: on both benchmarks SASA's average max toxicity and toxic rate are lower than RAD's.
Aligned models and gender bias
- Aligned model: Llama-3.1-8b-Instruct starts at average max toxicity 0.727 and toxic rate 0.892 on challenging RTP; SASA β=500 reduces these to 0.234 and 0.171 (Table 6). The authors state that aligned models' internal embedding spaces are more informative about risk attributes such as toxicity, and SASA's reduction on it (68%, 81%) is larger than on Llama-2-7b (51%, 54%).
- Gender bias: the BOLD gender domain has 2363 samples (776 American_actresses, 1587 American_actors, taking the first 776 male samples to balance), and Llama-2-7b has toxic rate 0.066 for the female group and 0.031 for the male group (Table 14); the authors state this suggests the LM is biased against women, and both RAD and SASA can balance the two groups' toxic rates, with SASA β=500 average max toxicity 0.025 versus RAD 0.049.
Other ablations and analyses
- Toxicity attribute breakdown (Table 9, challenging RTP, Llama-2-7b): SASA β=500 has Severe Toxicity 0.024, Identity Attack 0.017, Insult 0.181, Profanity 0.38, Threat 0.024; the authors state the probability of each attribute is lower than the corresponding RAD value.
- No-external-model baselines (Table 13, challenging RTP, GPT2-Large): SASA β=500 toxicity 0.267, perplexity 15.40; Self-Debiasing λ=100 toxicity 0.263, perplexity 23.548; ToxificationReversal toxicity 0.773, perplexity 45.780. The authors state the latter two cannot reach SASA's toxicity level at comparable perplexity.
- Word-filtering combination (Table 15, challenging RTP, Llama-2): SASA+403 banned-word filtering, at β=500 average max toxicity drops from 0.426 to 0.178 and perplexity rises from 7.20 to 19.66; the authors state toxicity decreases but generation coherence may be impaired.
- Runtime overhead: GPT2-Large original decoding 2.1 hours/3.1 GB, RAD 5.5 hours/4.3 GB, SASA 2.9 hours/3.8 GB; Llama-2-7b original 2.9 hours/15.5 GB, RAD 5.6 hours/16.7 GB, SASA 3.1 hours/17.3 GB; the authors note no complexity optimizations such as representation reuse were done.
- Sample efficiency (Figure 4): the authors state toxicity accuracy plateaus at around 500K samples, so training the classifier need not use all samples.
- Failure cases (Table 11): the authors state SASA often weakens language but sometimes retains some negative words, and note that whether certain samples judged toxic by Perspective API are actually toxic is disputed.
What are some points that can be further explored?
The authors state that SASA depends on the discriminative ability of the underlying LM embeddings, and combining multiple attributes is left for future work.
Limitations and future directions noted by the authors
- Dependence on the underlying LM's representational ability (Appendix A.7): SASA models the toxic subspace in the sentence embedding space; if the LM cannot capture and distinguish subtle attributes related to the target property, performance may suffer, especially for smaller or simpler models; the authors state that from Figure 2, the gap between SASA and RAD is smaller on Llama-2-7b and larger on GPT2.
- Multiple attribute constraints (Section 3.2): The authors state that SASA's modularity can accommodate multiple attribute constraints on the contextual embedding space, and combining multiple attribute constraints is left for future work.
- False positive/false negative factors (Appendix A.4): The authors point out two sources—the accuracy of the learned subspace is affected by the toxicity discrimination ability of the original LM embedding space, and the false positive rate of the Perspective API itself.
- Room for engineering optimization (Section 4.6): The runtime and memory data did not undergo complexity optimizations such as representation reuse; the authors state that more detailed engineering is needed to understand the limits of each decoding method.
Experimental coverage
- The models tested are GPT2-Large, Llama-2-7b, and Llama-3.1-8b-Instruct, 3 in total (762M–8B parameters).
- The benchmarks are RTP (10K non-toxic prompts and the challenging subset), BOLD (the first 1000 entries used in Table 4), and AttaQ; 25 generations per prompt.
- Toxicity determination all uses the Perspective API (threshold 0.5); the paper does not report human evaluation; the authors note that results from prior work may be based on different versions of the Perspective API, which changes over time.
- Fluency is assessed by the perplexity of a larger LM (GPT-2-XL or Llama-2-70b); the paper does not report human fluency evaluation.
- Subspace learning uses English comments from the Jigsaw dataset; the paper does not report experiments on data in other languages or domains.
Summarize the main content of the paper
SASA uses the LLM's own embeddings to learn a toxicity subspace that guides decoding, requiring no external reward model, with lower toxicity than RAD.
A paper on a decoding method for LLM detoxification, proposing SASA (Self-disciplined Autoregressive Sampling).
- Problem: existing detoxification methods either retrain the model or rely on an external reward model at inference time; the authors ask whether an LLM can detoxify itself using only its internal representations.
- Method: use toxicity-labeled data to analytically learn the Bayes-optimal linear classification subspace in the decoding LM's contextual embedding space; during decoding, compute per token the margin from the current context to the subspace and adjust the sampling distribution as p = Softmax(logit + β·πm); the authors prove this strategy is the optimal solution to a constrained optimization problem trading off the alignment objective against KL divergence from the reference distribution.
- Main results: on challenging RTP, Llama-2-7b's SASA β=500 has average max toxicity 0.426 and perplexity 7.20, versus RAD β=500 at 0.481 and 7.33 (Table 3); on AttaQ, SASA 0.142 versus RAD 0.264 (Table 5); on non-toxic RTP, GPT2-Large's SASA β=500 reaches 0.083 (Table 1).
- Other findings: on the aligned model Llama-3.1-8b-Instruct, SASA reduces average max toxicity from 0.727 to 0.234 (Table 6); combined with 403-word filtering it can drop further to 0.178 but perplexity rises to 19.66 (Table 15); the authors state SASA does not achieve a better toxicity-perplexity trade-off on all datasets (Figure 2).
- Authors' conclusion: LLMs can use their own contextual representations to learn a discriminative subspace for self-detoxification, and SASA achieves performance comparable to existing methods without an external reward model or retraining; a limitation is that effectiveness depends on the underlying LM embeddings' ability to distinguish toxicity, and combining multiple attributes is left for future work.