RLBF: Retrospective Feedback Reinforcement Learning for LLM Safety

Existing alignment tends to only refuse at the start of the query, lacking the ability to retract only the violating segment mid-generation and then continue answering.

• Scenario: The authors believe alignment is critical once models enter generation, reasoning, and other tasks; beyond harmful content under adversarial prompts, they also need to handle toxicity, bias, and misinformation.
• Limitation: The authors state that SFT, RLHF/RLAIF, and DPO tend to form "shallow safety," where refusals are mainly triggered by the initial tokens and can be bypassed by prefilling, GCG, and prompt injection. Reset discards usable later text; BSAFE requires repeating the harmful segment before rewriting, and the authors believe efficiency is still affected.
• Assumption: The authors believe the model should directly emit "backtrack by x tokens," retracting only to the safe state before the violation, to reduce rewriting traces.
• Approach: RLBF uses BSAFE+ to inject violations into coherent safe text for SFT, then uses a critic to provide rewards on real-time outputs, learning to backtrack and continue generation.
• Threat model:
  • Goal: Reduce harmful generation under adversarial attacks and in-distribution errors, measured by ASR.
  • Knowledge: The paper does not describe attacker knowledge as white-box, black-box, or gray-box.
  • Capabilities: Evaluations include Middle Filling (injecting malicious instructions after benign context), GCG, Decoding Parameters, and an adaptive GCG from Zhang et al. [2025] that optimizes suffixes to suppress backtracking tokens. The authors state training does not include GCG-style or decoding-exploit samples.
  • Victim system: Gemma 2 and LLaMA 3. At inference, the model autoregressively emits category tokens and [BACKTRACK_BY_X], and post-processing deletes the X tokens before it; the paper states the KV cache is not rolled back.

The paper's Related Work is divided into safety alignment and generation refinement; Section 3 then compares reset and BSAFE.

Safety alignment

• RLHF/RLAIF: Ouyang et al. [2022], Bai et al. [2022a,b] train reward models with human or AI feedback, then fine-tune with PPO, pursuing helpful and harmless. The authors state RL-based methods are computationally expensive and complex to implement.
• Direct fine-tuning: DPO [Rafailov et al., 2023], as well as non-RL safety techniques represented by Yuan et al. [2023]. The authors state many existing alignments still exhibit shallow safety, which can be bypassed by attacks that manipulate prompt structure or inject instructions later in the input, citing Qi et al. [2025], Zhang et al. [2025], Carlini et al. [2023].

Generation refinement and self-correction

• Iterative rewriting: Self-refine [Madaan et al., 2023] and others iteratively improve outputs using multiple perspectives or alternative continuations; the paper also cites Ma et al. [2023], Sel et al. [2024b]. Long [2023], Yao et al. [2023], Sel et al. [2024a] incorporate exploration and refinement into the generation process. Here the paper only describes them.
• Modifying the process after detecting unsafe content: reset [Qi et al., 2025; Zhang et al., 2025], targeting suffix attacks [Zou et al., 2023b], adjusting decoding parameters [Huang et al., 2024], adaptive jailbreaks [Andriushchenko et al., 2025]. Circuit Breakers [Zou et al., 2024] control internal representations, interrupting the model when harmful output is imminent.

Backtracking mechanisms

• Reset: Return to the start of generation, or emit [RESET], often combined with masked SFT, sometimes with DPO. The authors state it suits problems early in the sequence; when violations are deep, it may discard many tokens to fix a small segment.
• BSAFE [Sel et al., 2025b]: Emit a category token (e.g., [TOXICITY], [HEALTH_VIOLATION]), replace the harmful segment with a safe alternative, then continue; backtracking probability can be controlled by category at test time. The authors state rewriting still harms efficiency. Its data has one model answer questions and another label them; the authors state the examples are generic and lower quality, and single-model continuations may be out-of-distribution. BSAFE's authors did not report a drop on math benchmarks. The authors use Gemini 2.0 to judge on LMSYS: an IT trained with that data policy wins 28.2%, while standard IT wins 71.8%.

Baselines and positioning

• Baseline methods: IT, RL without the backtracking-specific reward, Circuit Breakers, BSAFE+. Benchmarks are the LMSYS harmful query subset, LMSYS-MF, AdvBench, HEx-PHI, MaliciousInstruct, as well as MMLU, BBH, GSM8K, MATH.
• The authors distinguish themselves on three points: using [BACKTRACK_BY_X] instead of full reset or repeated rewriting; BSAFE+ injecting violations into safe responses from the model being trained, making the backtracking position and safe continuation known and in-distribution (the authors write the quality as 49.4% vs. 50.6%, without identifying which side is which); and the RL objective optimizing based on the critic's dynamic feedback over the whole generation, rather than only maximizing the likelihood of control tokens and replacement text on static data.

RLBF has two stages: SFT teaches the category tokens and [BACKTRACK_BY_X], then a single LLM critic's trajectory reward and GRPO update the policy. In Figure 1, the left column shows the query, IT response, attacker rewrite, and BSAFE+ samples with [BACKTRACK]; the right column shows RLBF generation, the critic, and post-backtrack continuation; the sentences in the figure are not repeated. The introduction states the critic can be specialized by safety category; Section 4.2.1 actually uses a single critic.

Backtracking signal and inference-time handling

• When a violation spans X tokens and ends at yk, the model first emits [CATEGORYc], then [BACKTRACK_BY_X], where X is a positive integer.
• Inference remains autoregressive, without rolling back the KV cache. Special tokens are handled by post-processing or a streaming handler: delete the X tokens before the signal from the user-facing output stream, then append the tokens after the signal. The authors state a small buffer can enable near-real-time correction, avoiding regenerating harmful content and BSAFE's replacement sequences.
• The authors state backtracking probability can still be controlled by category at test time. The paper does not report experimental numbers for this control. The critic is used for training rewards and SFT data analysis; the paper does not state that the critic is called at inference.

BSAFE+ and SFT

• First, the instruction model to be trained generates safe responses to relevant queries (e.g., chat data), then harmful or jailbreak segments are injected at contextually coherent positions. Because the full safe response is known, the backtracking position and safe continuation are known, and the continuation is in-distribution for that base model.
• The SFT input is the prompt followed by the safe prefix. The targets are the category token, the backtracking command, and the original safe latter half (with minor fluency adjustments). X is determined by the violating segment length and the immediately preceding, flagged-as-violating context. The loss is standard cross-entropy. SFT mixes ordinary instruction data; the paper does not state the quantity. Section 5 states the augmented samples are 300k.
• Appendix A.1 is the harmful continuation module: it takes the user query and assistant prefix as input and outputs a tagged continuation with a category. Appendix A.1 provides the full prompt.

Critic and rewards

• A single LLM critic monitors y1 through yk. When it detects a violation, it gives a category and token span, yielding X.
• 100 pairs are manually constructed, with severity 43 high, 38 medium, 19 low. The three candidate critic prompts achieve accuracies of 97.6%, 97.6%, and 98.0% (three samples per pair). The authors state it is insensitive to prompt wording. The critic module in Appendix A.2 takes the user prompt and assistant response as input and outputs whether there is a violation and the minimal edit; Appendix A.2 provides the full prompt.
• R_final is given only at the end of the trajectory. Without a backtracking signal: any violation is -1.0, and no violation throughout is +1.0. When a signal is present, the user sees the retained prefix followed by the suffix after the signal. An unnecessary backtrack (the deleted segment was actually not a violation) is -0.5. A necessary backtrack where the critic judges the result safe, coherent, and useful is +1.0; unsafe, incoherent, or useless is -0.2. The authors state this encourages directly generating safe content, backtracking appropriately on errors, and ensuring correction quality.

GRPO and SFT guidance

• The policy is optimized with GRPO [Shao et al., 2024]. The core objective is to maximize the expected trajectory reward: J_RL(θ)=𝔼_τ∼πθ[R_final(τ)]. The total loss is: L_total(θ)=−J_RL(θ)+λ_SFT L_{SFT_guidance}(θ). The former is the RL objective; the latter treats the log-likelihood on SFT samples as behavior cloning; λ_SFT=0.1.
• Learning rate is 1e-5 for SFT and 5e-6 for RL; AdamW, 100-step linear warmup then cosine decay; bf16; A100 80GB. The authors state about 800 GPU hours per model (SFT 250, RL 550). The paper does not report GRPO's group size or KL coefficient.
• If a rollout regenerates a known violation v that was replaced in SFT, the authors write it can be counted as a strong penalty in R_final, or the probability of such sequences can be penalized in the GRPO update. The paper does not give a coefficient for this penalty.
• The authors compare with BSAFE: the latter mainly maximizes the likelihood of control tokens and replacement text on static data; J_RL considers dynamic feedback over the whole generation, including whether the output after backtracking is safe and useful. The authors state the feedback comes from the model's own generation, targets in-distribution failures, and trains recovery rather than only prevention.

Experimental Setup

• Models under test: Gemma 2 2B, Gemma 2 9B, LLaMA 3 1B, LLaMA 3 3B, LLaMA 3 8B. Some numbers in Table 2, Table 5, and Section 5.6 do not specify the model.
• Baselines: IT; RL, i.e., trained with the authors' reward but without the backtracking-specific reward; Circuit Breakers; BSAFE+.
• Benchmarks: LMSYS harmful query subset (Table 1 row named LMSYS); LMSYS-MF; GCG uses AdvBench and HEx-PHI; Decoding Parameters uses MaliciousInstruct and HEx-PHI; adaptive GCG from Zhang et al. [2025]. Utility is MMLU, BBH, GSM8K, MATH. The sample sizes of each attack set are not reported in the paper.
• Metrics: ASR; Table 4 is Attack Prevention Rate; Table 3 is Solution Rate. The review model and threshold for ASR are not specified in the paper.
• Training and critic: 300k SFT samples, λ_SFT=0.1, learning rate and hardware see Question 3. The model name of the stronger critic is not specified in the paper. The weaker critic is LLaMA 3 8B (same name as the tested 8B), same prompt, validation set accuracy 82.0%.
• Other reviews: Section 3 uses Gemini 2.0 to judge the LMSYS win rate of the BSAFE data strategy. The number of repetitions for the main table's ASR is not reported in the paper.

Main Results

Swipe horizontally to read the table

Model	MF IT	MF BSAFE+	MF RLBF	LMSYS BSAFE+	LMSYS RLBF
Gemma 2 2B	71	4	5	14	2
Gemma 2 9B	75	3	3	15	2
LLaMA 3 1B	68	6	7	14	1
LLaMA 3 3B	77	5	5	17	2
LLaMA 3 8B	81	5	3	16	1

Units are ASR%, according to Table 1. RL and Circuit Breakers are not listed in this table.

• The authors state that IT's high ASR corresponds to shallow safety. The authors give RL as LMSYS-MF 61%–72%, LMSYS 22%–25%; CB as MF 13%–18%, LMSYS 17%–21%, and state that CB outputs incoherent text after triggering.
• The authors state that on MF, BSAFE+ is 3%–6% and RLBF is 3%–7%, and the two are close. In Table 1, not every row has RLBF lower than BSAFE+: Gemma 2 2B and LLaMA 3 1B are the opposite.
• The authors state that on the standard LMSYS harmful subset, RLBF is lower than BSAFE+. The authors hypothesize the reason is: RL simultaneously optimizes generating fewer violations and correcting via backtracking; "backtrack by x tokens" is easier to learn than BSAFE's multi-token replacement sequence. This is a hypothesis, not a separate experiment.

GCG, Decoding Parameters, and Adaptive Attacks

Swipe horizontally to read the table

Attack	Benchmark	IT	RL	CB	BSAFE+	RLBF
GCG	AdvBench	65.6	62.6	10.7	6.6	4.7
GCG	HEx-PHI	36.5	38.3	13.4	5.7	4.3
Decoding Parameters	MaliciousInstruct	84.3	81.8	2.0	1.0	1.0
Decoding Parameters	HEx-PHI	54.9	51.7	12.4	5.0	3.7

Units are ASR%, according to Table 2. The paper does not specify which model this corresponds to.

• The authors state that RLBF has the lowest ASR on GCG, giving 4.3%–4.7%. On HEx-PHI with GCG, RL is higher than IT. On MaliciousInstruct, BSAFE+ and RLBF are both 1.0%.
• The authors state that neither the BSAFE+ data nor the RL stage contains GCG-style or decoding-exploit samples.
• The adaptive GCG also optimizes the suffix to suppress backtracking tokens. The authors state that RLBF's ASR is unchanged, still AdvBench 4.7%, HEx-PHI 4.3%; BSAFE+ changes by at most 0.4 points.

Utility

Swipe horizontally to read the table

Model and Method	MMLU	BBH	GSM8K	MATH
Gemma2 9B IT	70.6	67.4	66.4	35.4
Gemma2 9B RLBF	70.7	67.3	66.3	35.6
LLaMA 3 8B IT	66.3	64.1	63.1	49.8
LLaMA 3 8B RLBF	66.6	64.2	63.1	49.9

Units are Solution Rate%, according to Table 3. Utility for smaller models is not reported in the paper.

• BSAFE+ for Gemma2 9B is 70.1, 67.1, 66.3, 35.4; for LLaMA 3 8B is 66.7, 63.8, 63.2, 49.6 (Table 3).
• The authors state that safety enhancement does not come at the cost of utility; backtracking is mainly invoked when the critic detects a violation, and during benign generation the model still works as an instruction model. The authors state that mixing instruction data and the RL objective avoids catastrophic forgetting.

Backtracking Ablation and Critic Quality

• Table 5 compares backtracking on LMSYS: IT 24, RL 22, BSAFE+ 14, RLBF 1, removing all backtracking signals 18, removing mid-generation backtracking 7 (ASR%). The paper does not specify the model; the first four columns are the same as one LMSYS row in Table 1. The authors state that mid-generation backtracking is also important.
• Section 5.5 redoes RL with a LLaMA 3 8B critic with 82.0% accuracy. On LMSYS, ASR is 2%–4%, with the stronger critic 1%–2%, and BSAFE+ 14%–17%. On LMSYS-MF, both critics are 3%–7%.
• The authors state that the gain comes from policy learning with backtracking, not just from critic accuracy.

Other Ablations and Analyses

• Table 4: Attack Prevention Rate of RLBF on LMSYS-MF, models are Gemma 2 2B, LLaMA 3 1B, LLaMA 3 3B. Hate Speech is 0.98/0.96/0.96, Sexually Explicit is 0.92/0.94/0.92, Illicit Drugs is 0.94/0.92/0.92. The authors state that Hate Speech, Toxic, Politics, Health, Violent, Finance are generally at least 0.96; in Table 4, LLaMA 3 3B's Finance is 0.94.
• Section 5.6: Using BSAFE-style backtracking on the same data, LMSYS ASR is 17%–22%, between RL and BSAFE+. The authors state that the data construction in Section 4.1.2 also has an effect independent of the backtracking format.
• On responses that trigger backtracking, BSAFE+ and RLBF average 68.3 and 72.6 fewer tokens than BSAFE; when the same correction is replayed in BSAFE format, it is 74.8 and 79.1. The authors state that at typical service speeds this corresponds to 2.5–3.5 s less per correction for hosted models and 5–10 s less for edge devices. The paper does not report measured latency.
• Section 3: IT with the BSAFE data strategy is judged by Gemini 2.0, with an LMSYS win rate of 28.2%, and standard IT is 71.8%. The authors write BSAFE+'s answer quality as 49.4% vs. 50.6%, without specifying which side is which, and without stating whether it is still judged by Gemini 2.0.

Limitations and Future Directions Noted by the Authors

• Computational requirements: The authors write that RLBF's computational requirements, especially the backtracking process, may pose challenges in some deployment scenarios (Section 7).
• Definition of harmful: The authors write that precisely defining "harmful" has inherent difficulties, and existing safety protocols may not cover every potential violation (Section 7).
• Future directions: The authors suggest future research on more adaptable strategies to handle this complexity (Section 7).

Experimental Coverage

• Table 1 reports 5 models: Gemma 2 2B, 9B and LLaMA 3 1B, 3B, 8B, comparing IT, RL, Circuit Breakers, BSAFE+, RLBF, on the LMSYS-MF and LMSYS harmful query subset benchmarks.
• Section 5.1 additionally reports GCG (AdvBench, HEx-PHI), Decoding Parameters (MaliciousInstruct, HEx-PHI), and adaptive GCG from Zhang et al. [2025] that optimizes the suffix to suppress backtracking tokens. Table 2 does not specify the model.
• Table 3's utility only includes Gemma2 9B and LLaMA 3 8B, on the MMLU, BBH, GSM8K, MATH benchmarks.
• Section 5.5 compares a LLaMA 3 8B critic with 82.0% accuracy and a stronger critic; the model name of the stronger critic is not specified in the paper. The review model for ASR, the sample sizes of each attack set, and the number of repetitions for the main experiments are not reported in the paper.
• Section 4.2.1 uses 100 manually constructed critic prompt pairs for validation, with severity 43 high, 38 medium, 19 low, each pair sampled three times.

RLBF is a self-correction safety training method during generation, targeting what the authors call shallow safety, as well as the issue of reset and BSAFE discarding too many tokens when retracting violations.

• Problem: Refusals are often triggered by the query's initial tokens, and prefilling, GCG, and prompt injection can bypass them. If a violation appears later, resetting the whole segment or repeating before rewriting discards more generated text than necessary.
• Method: BSAFE+ injects violations into coherent safe responses, and SFT learns [CATEGORYc] and [BACKTRACK_BY_X]. A single LLM critic gives trajectory rewards from -1.0 to +1.0 based on violations, unnecessary backtracks, and correction quality, then GRPO with SFT guidance at λ_SFT=0.1 updates the policy. Inference does not roll back the KV cache; post-processing deletes the X tokens before the signal.
• LMSYS: On the harmful query subset in Table 1 across five models, RLBF's ASR is 1%, 2%, 1%, 2%, 1%, while BSAFE+ is 14%–17%. On MF, the authors state RLBF is 3%–7% and BSAFE+ is 3%–6%; on Gemma 2 2B and LLaMA 3 1B, RLBF is not lower than BSAFE+.
• Other attacks and utility: In Table 2 (model not stated), GCG's RLBF is 4.7% on AdvBench and 4.3% on HEx-PHI; under adaptive GCG, the authors state these two numbers are unchanged. On MaliciousInstruct under Decoding Parameters, BSAFE+ and RLBF are both 1.0%, and on HEx-PHI, RLBF is 3.7%. Gemma2 9B's RLBF is 70.7% on MMLU and 70.6% on IT (Table 3).
• Authors' conclusion: The authors state dynamic self-correction reduces ASR across multiple models and benchmarks while maintaining utility; they also believe fixing errors in-distribution helps more against unseen attacks than only preventing errors. Section 7 also discusses computational requirements and that harmful is difficult to define precisely.