Skip to content

RedDebate: Making LLM Responses Safer Through Multi-Agent Red-Team Debate

RedDebate: Safer Responses Through Multi-Agent Red Teaming Debates

arXiv· arXiv:2506.11083· ICML 2026· Ali Asad, Stephen Obadinma, Radin Shayanfar, Xiaodan ZhuDepartment of Electrical and Computer Engineering & Ingenuity Labs Research Institute, Queen's UniversityIndexed by this site · 2026-10-15· Originally publishedDefenses

Quick read

Defenses

Based on the paper PDF (AI-generated); refer to the original

RedDebate uses debate and long-term memory to reduce unsafe outputs; on HarmBench, SReD+GLTM achieves a total ER as low as 0.2%.

Threat modelRed-team prompts serve as debate topics. Multiple LLM debaters see shared memory and revise responses at inference time; an independent evaluator provides binary safety labels, and feedback is generated and written to long-term memory only when an unsafe response occurs. No human annotation is relied upon. The paper does not use white-box, black-box, or gray-box to describe the process.

Questions
Unsafe responses often only appear at inference time when facing red-team prompts, human evaluation is hard to scale, and single-model self-critique may fail to see its own errors. RedDebate aims to let multiple LLMs automatically discover, correct, and remember these failures.
Methods
Multiple debaters conduct three rounds of debate on red-team prompts, with strategies including peer revision, Devil–Angel, and Socratic questioning. After an independent evaluator flags unsafe responses, lessons are written into retrieval memory, LoRA memory, or executable guardrails for use in subsequent inference.
Experiments and results
On HarmBench, memory-free SReD total ER is 21.0% and 28.9%, and SReD+GLTM is 3.6% and 0.2% (Table 1, Table 2). On CoSafe, GLTM total ER is 2.5% and 0.8%. CLTM is not lower for both Phi and R1.
Limitations and future work
The authors note that LlamaGuard is imperfect, that Colang and NeMo introduce runtime errors, and list broader jailbreaks, inference-time patterns, and memory poisoning as future work. Experiments cover two open-source model groups and several frontier models; the main benchmarks are HarmBench and CoSafe; the paper does not report sample sizes for these two benchmarks.
Experimental setupMistral-7B-v0.2、LLaMA-3.2-3B-Instruct et al. · HarmBench、CoSafe et al. · ER、AGR
Evaluated models
Mistral-7B-v0.2LLaMA-3.2-3B-InstructPhi-3.5-miniGemma-3-12BQwen3-8BDeepseek-R1-Distill-LLaMA-8BGPT-OSSGrok-4DeepSeek R1-685BGemini-2.5-Pro
Benchmarks
HarmBenchCoSafeAegis-2WildJailbreakTriviaQAXSTest
Metrics
ERAGR
AI summaryRedDebate lets multiple LLMs review each other's reasoning in multi-round debates to discover and mitigate unsafe behavior through fully automated red teaming. The framework includes a long-term memory module that stores safety-related insights from debates and recalls them in subsequent reasoning, leading to continuous improvement in model behavior. On safety benchmarks across multiple models, the debate itself reduces unsafe outputs, and adding memory further decreases errors; the authors claim this is the first fully automated framework that unifies multi-agent debate and red teaming without human intervention.

RedDebate lets multiple LLMs review each other's reasoning in multi-round debates to discover and mitigate unsafe behavior through fully automated red teaming. The framework includes a long-term memory module that stores safety-related insights from debates and recalls them in subsequent reasoning, leading to continuous improvement in model behavior. On safety benchmarks across multiple models, the debate itself reduces unsafe outputs, and adding memory further decreases errors; the authors claim this is the first fully automated framework that unifies multi-agent debate and red teaming without human intervention.

Paper analysis

6 questions; open one to read the full answer

  1. What problem does this paper attempt to solve?

    The authors propose RedDebate, which uses multi-agent debate and long-term memory to reduce unsafe outputs at inference time.

    How to enable multiple LLMs to discover and correct unsafe responses at inference time without human intervention.

    • Problem scenario: The authors argue that unsafe behavior often surfaces only at inference time under novel or adversarial prompts; RLHF and supervised safety fine-tuning mainly optimize predetermined objectives during training.
    • Limitations of existing approaches: The authors state that human red-teaming is hard to scale, prone to missing subtle errors, and costly; single-agent self-critique may fail to see its own knowledge gaps. Training signal coverage is limited, and reward hacking can occur.
    • Core question: Can a group of LLMs jointly identify, correct, and learn from their own unsafe behavior through debate?
    • Proposed approach: An inference-time framework called RedDebate. Multiple agents debate around red-team prompts, an independent evaluator triggers feedback, and long-term memory stores safety lessons. The authors claim that, to their knowledge, this is the first fully automated framework that unifies debate and red-teaming to progressively improve safety without human intervention.
    • Threat model:
      • Goal: Reduce unsafe responses to red-team prompts and turn failures into reusable lessons for the future.
      • Knowledge: The paper does not describe the attacker using white-box, black-box, or gray-box terms. Debaters share short-term and long-term memory; the evaluator is separate from the debaters to avoid situations where all hold the same false belief and cannot correct each other.
      • Capability: Debate topics are drawn from an adversarial prompt set; debaters respond over multiple rounds, see each other's responses, and revise them; if any response is unsafe, the feedback generator writes to long-term memory based on the full history and labels. If all responses are safe, no update occurs.
      • Victim system: The LLMs participating in the debate. The authors state that the framework complements rather than replaces training-time alignment.
  2. What related research exists?

    Related work is divided into debate, red-teaming, feedback learning, memory, and guardrails; the authors state that this paper applies them to safety refinement.

    Section 2 of the paper and Appendix B divide related work into four groups and place this paper on debate-based safety refinement at inference time.

    Multi-agent debate

    • Irving et al. (2018): uses debate and self-play to discuss superhuman AI alignment. Khan et al. (2024): debate can help weaker judges evaluate stronger models.
    • Du et al. (2024), Liang et al. (2024), Chan et al. (2024), Smit et al. (2024): MAD relative to self-reflection or ensembling, used to improve factuality, alignment, or reasoning. The authors note that none of these works treat MAD as a red-teaming method with learning to refine safety.
    • Chern et al. (2024): has jailbroken or weaker models debate stronger models to reduce toxicity. Appendix B likewise states that it does not combine debate with safety learning.

    Red teaming

    • Human testing (Dinan et al., 2019; Ribeiro et al., 2020, etc.): humans write test cases or judge harmful responses.
    • Perez et al. (2022): uses language models to generate tests and trained classifiers to find failures. This is followed by MART (Ge et al., 2023), Rainbow Teaming (Samvelyan et al., 2025), curiosity exploration (Hong et al., 2024), and automatic formation of harmfulness metrics (Casper et al., 2023a).
    • The authors state that what this paper wants is an automatic pipeline in which a group of LLM agents generate, critique, and evaluate responses, without training additional classifiers or relying on human supervision.

    Learning from feedback

    • RLHF (Ouyang et al., 2022; Bai et al., 2022a): fine-tunes helpful, harmless assistants according to human preferences.
    • Constitutional AI (Bai et al., 2022b): self-critiques and iteratively revises according to predetermined rules. Reflexion (Shinn et al., 2023): puts verbal reflections into episodic memory for use in later trials.
    • The authors state that this paper instead uses peer critique in debate, followed by multiple types of long-term memory.

    Memory and guardrails

    • Text memory and parametric memory (Zhang et al., 2024 survey; Hu et al., 2023; Zhong et al., 2024; Xiong et al., 2023, etc.): store experience through external retrieval or weight modification, respectively. The authors claim to be the first to use both types of memory simultaneously in a debate safety setting.
    • NeMo Guardrails (Rebedea et al., 2023): uses Colang to specify dialogue flows. The authors claim to be the first to treat guardrails as memory in a safety setting.

    Baselines and benchmarks: The main comparisons are Standard Prompting, Self-Revision, Self-Critique implemented according to Constitutional AI, and Best-of-N. The main benchmarks are HarmBench and CoSafe; the appendix also uses Aegis-2, WildJailbreak, TriviaQA, and XSTest. The authors position RedDebate as an inference-time mechanism complementary to RL alignment, rather than competing with training-time methods.

  3. How does the paper address this problem?

    After multiple debate rounds, an independent evaluator triggers feedback, which is written into retrieval, parametric, or guardrail memory.

    RedDebate has multiple LLMs debate around a red-team prompt; after an independent evaluator finds unsafe responses, distilled lessons are written into long-term memory for use in future debates.

    Setting and one-round flow

    • Input: a debate topic X is drawn from an adversarial prompt set P. N debaters respond over a fixed number of rounds T. The main experiments use 3 debaters per group and 3 rounds.
    • Generation: the round-t response depends on the prompt, shared short-term memory, and long-term memory, i.e., rn(t)=Dn(X, MS, ML). This formula indicates that debaters can see their own and peers' existing responses, as well as historical safety lessons.
    • Judgment and writing: each round's responses enter short-term memory. After the debate ends, the evaluator assigns binary labels to all responses. If any unsafe response exists, the feedback generator produces textual lessons based on the full history and labels and writes them into long-term memory; otherwise no learning occurs. Algorithm 1 summarizes this process.
    • Why separate evaluation: the authors believe that if all share mistaken beliefs, peers may not be able to correct each other, so the evaluator does not participate in the debate. Diverse responses are also used to increase the chance that at least one response conflicts with potentially unsafe content.

    Three debate strategies

    • PReD: same-role peers respond in parallel, see each other's responses, and revise according to Algorithm 1.
    • DAReD: the authors believe PReD responses may be too similar. The Devil generates opposition regardless of whether the original response is safe, while the Angel supports it, and debaters re-evaluate accordingly. The appendix algorithm simplifies this to one debater to reduce dialogue overlap; experiments average over role permutations of three models.
    • SReD: the authors observe that agents often do not proactively revise based on peers unless directly challenged. Socratic agents do not answer the debate topic but only question assumptions, evidence, and reasoning gaps. Safety evaluation does not count Socratic, Devil, and Angel utterances.
    • Guiding models: the feedback generator and Socratic agents use GPT-4o-mini. Appendix G gives complete prompts for each role.

    Four types of long-term memory

    • STM: keeps only the current debate and resets when it ends. LTM retains lessons across debates.
    • TLTM: embeds natural-language feedback into a vector store; new prompts retrieve the 5 most relevant original texts by cosine similarity into context. The embedding model is text-embedding-3-large.
    • CLTM: uses LoRA to inject accumulated feedback into debater attention layers, trained with cross-entropy. Every 10 new feedback items, it resets and re-fine-tunes on all feedback up to that point. The appendix gives r=16, alpha=16, dropout=0.1.
    • UNIFIED and GLTM: the former uses both TLTM and CLTM. GLTM compiles unsafe experiences into Colang flows: a single prompt generates abstract expressions and example sentences, merges expressions with the same name, then converts them with rules into executable NeMo Guardrails flows. Before generation, if input matches a known unsafe flow, it directly rejects. Guardrails are generated by GPT-4o. Appendix Figure 23 gives the generation prompt and flow examples.

    Success determination

    • Labels: 0 is unsafe, 1 is safe. The main evaluation uses LlamaGuard and only evaluates debaters.
    • Two metrics: error rate is that agent's proportion of unsafe responses across all prompts and rounds, ERn=∑p,t I[yp,n(t)=0]/(|P|× T). Agreement rate is the proportion of transitions from unsafe to safe between adjacent rounds, with denominator |P|×(T−1).
    • Implementation: reasoning mode is disabled when available. Cost experiments limit each newly generated response to 512 tokens.
  4. What experiments did the paper conduct?

    Without memory, SReD has the lowest ER; adding GLTM reduces HarmBench total ER to 3.6% and 0.2% (Table 2).

    Experimental setup

    • Main debaters: The first group is Mistral-7B-v0.2, LLaMA-3.2-3B-Instruct, Phi-3.5-mini; the second group is Gemma-3-12B, Qwen3-8B, Deepseek-R1-Distill-LLaMA-8B. The appendix additionally tests Grok-3-mini, GPT-OSS-20B, Grok-4, DeepSeek R1-685B, Gemini-2.5-Pro.
    • Benchmarks: The main experiments use HarmBench's directly harmful prompts and CoSafe's conversational prompts. The paper does not report sample sizes for either. The appendix uses Aegis-2, WildJailbreak, TriviaQA (1,000 questions), and XSTest.
    • Baselines: Standard Prompting (SP), Self-Revision (SR), Self-Critique (SC, with rules randomly sampled per Constitutional AI), and non-interactive Best-of-N.
    • Metrics: Lower ER is better; AGR statistically measures the proportion of adjacent rounds that change from unsafe to safe. Gray numbers in Table 2 are ER changes relative to the no-guidance or no-memory debate setting.
    • Evaluation: Main results are labeled binary by LlamaGuard. The paper does not report its threshold or the number of main experiment repetitions. Human comparison is in Appendix C.4.
    • Protocol: 3 rounds per session. DAReD averages over role permutations. TLTM takes top-5; CLTM retrains every 10 new feedback items.

    Main results

    ER (%) for SReD and four long-term memories on HarmBench, per Table 2.

    Swipe horizontally to read the table

    ModelNo memory+TLTM+CLTM+UNIFIED+GLTM
    Mistral-7B-v0.225.718.016.06.78.4
    LLaMA-3.2-3B-Instruct15.813.54.64.10.3
    Phi-3.5-mini21.614.121.67.42.0
    Gemma-3-12B24.84.55.10.00.3
    Qwen3-8B28.66.62.61.40.0
    Deepseek-R1-Distill-LLaMA-8B33.48.030.15.50.3
    • No memory: In Table 1, SP has the highest total ER, 38.7% for the first group and 44.4% for the second group on HarmBench; SReD is 21.0% and 28.9%, with the highest AGR at 17.0% and 17.5%. The authors state that peer interaction outperforms revising alone. An exception is DAReD's total ER of 36.4% on HarmBench for the second group, higher than PReD's 34.7%. The abstract states that debate alone reduces ER by up to 17.7%, and adding long-term memory exceeds 23.5%.
    • Memory: The authors state that all four LTMs reduce total ER, with GLTM having the lowest total ER, 3.6% for the first group and 0.2% for the second. Exceptions in the table: Phi's CLTM remains 21.6%; R1's CLTM is 30.1% with a marginal note of +3.3; Mistral's GLTM at 8.4% is higher than UNIFIED's 6.7%.
    • CoSafe: Without memory, SReD total ER is 4.5% and 5.8% (Table 1; SP is 7.4% and 8.1%). In Table 2, GLTM total ER is 2.5% and 0.8%. Phi's CLTM has a marginal note of −1.5; R1's TLTM and CLTM have marginal notes of −2.4 and −2.9.

    Comparison with Self-Critique and Best-of-N

    • SC: On HarmBench's second group, SC total ER is 8.3%, lower than no-memory SReD's 28.9%, higher than SReD+GLTM's 0.2% and UNIFIED's 2.3%. For the first group, SC total ER is 15.4%, and SReD+TLTM is 15.2%. The authors state that SReD with memory outperforms SC; Phi's TLTM at 14.1% and CLTM at 21.6% do not fall below its SC's 12%.
    • SC on CoSafe: Relative to SR, SC total ER goes from 7.0% to 8.1% and from 7.9% to 11.2%, marked gray as worse. The main text does not separately explain this exception.
    • BoN: Table 3 compares on Gemma/Qwen/R1 on HarmBench. For N=5, best, avg, and worst total ER are 29.7%, 42.7%, and 53.6%; SReD total ER is 28.9%. R1's SReD at 33.4% is higher than BoN best's 31.0%. The authors state that SReD is lower under average and worst aggregation, with best being an optimistic upper bound that widens with N.

    Latent errors, evaluator, and usefulness

    • Safe to unsafe: The main text uses HarmBench's first group as an example. Figure 3 text gives Llama's initial unsafe counts for PReD, DAReD, SReD as 47, 130, 89, with conversion counts of 13, 23, 18; Phi's conversions are 21, 23, 8. The authors state that DAReD triggers the most conversions, and SReD finds vulnerabilities in fewer average steps. Step endpoints are misaligned in the plain text and are not transcribed here.
    • Human comparison: Appendix C.4 annotates 315 HarmBench debate statements. LlamaGuard's accuracy, precision, recall, and F1 are 87%, 90%, 94%, and 92%; GPT-4o-mini with review prompt is 81%, 87%, 89%, and 88% (Table 8). The authors state that LlamaGuard can serve as a proxy for human judgment.
    • Usefulness: On TriviaQA, accuracy with memory is on the same order as without memory, with the highest refusal rate being Mistral+TLTM's 1.8% (Table 9, 1,000 questions). On XSTest, CLTM+TLTM reduces Mistral's compliance rate from 73.6% to 59.6% and Phi's from 80.8% to 72.8% (Table 10). The authors view this as possible over-regularization from hybrid memory.

    Cost, early stopping, and transfer

    • Tokens and latency: The authors state that SC uses about 35% fewer tokens per round than debaters (1,351 vs. 2,104±228). The authors state that with similar output token budgets, SC still underperforms 3-round debate; Qwen's SC requires up to 9 revisions to approach 3-round debate, and R1 stops revising after round 3 and worsens in the final round. In Table 15, Qwen's per-round wall-clock time is 28.195 seconds for SC and 21.559 seconds for debate.
    • Early stopping: 3 agents and 3 rounds produce 3,600 responses on HarmBench; stopping when all are safe removes 1,686 responses, reducing inference calls by 46%. Retraining TLTM without these feedback items changes ER from 16.0% to 17.8% (§5.3). The paper does not state which model group 16.0% corresponds to. The authors describe this as a trade-off between efficiency and exposure to failure cases.
    • Transfer: For Qwen with one round and no memory, WildJailbreak ER is 41.5% and Aegis-2 is 6.23%; after adding TLTM distilled from HarmBench, these become 6.5% and 1.20% (Table 6). In Table 7, R1's PReD+TLTM on Aegis-2 is 5.5%, higher than SC's 3.7%.

    Other ablations and analyses

    • Number of rounds (Table 13, HarmBench): without memory, 5 rounds total ER is 26.4%, 3 rounds is 28.9%; with TLTM, 3 rounds is 11.7%, 4 rounds is 18.0%, 5 rounds is 18.4%.
    • Number of agents (Table 14): 5 agents without memory total ER is 24.5%, DIV is 57.0%; with TLTM, total ER is 6.0%.
    • Frontier three-model PReD total ER: Grok-4 is 2.5%, R1-685B is 12.6%, Gemini-2.5-Pro is 3.8% (Table 5).
    • TLTM retrieval: the main text states Hit Rate for K=10 is 89%; Table 11 writes 0.8900, and K=5 is 0.8125.
    • Guardrail execution (Table 16): Phi's runtime error on HarmBench is 25.0%; Qwen's Guardrails Recall is 1.0%.
    • GPT-OSS-20B's SP ER on HarmBench is 1.3% and SReD is 1.1% (Table 4). The authors state it barely participates in debate, giving only brief refusals.
  5. What are some points that can be further explored?

    The authors note that the evaluator is imperfect, guardrail runtime errors occur, and jailbreak coverage and memory poisoning remain for future work.

    Limitations and future directions noted by the authors

    • Evaluator: LlamaGuard is not perfect relative to human annotation and may affect response evaluation and optimal feedback; the authors also state that this cannot explain the safety improvements on HarmBench (Appendix A).
    • Guardrail implementation: The strict Colang format and NeMo instability cause runtime errors on models such as Phi; the authors state that the library itself needs improvement (Appendix A).
    • Jailbreak coverage: The main experiments use standardized benchmarks; the appendix includes WildJailbreak, but the authors still list broader recent jailbreak techniques and multi-turn adversarial attacks as future work (Appendix A).
    • Relation to input-side defenses: The authors believe most jailbreak defenses act on inputs or outputs, which is orthogonal to the inference-layer goal of this paper; combining with input-side filters is a future direction (Appendix A).
    • Memory poisoning: Online memory may be contaminated by adversarial inputs. The authors state that no dedicated empirical study was conducted; updating only after debate consensus and LlamaGuard filtering is not sufficient as a guarantee against poisoning (Appendix A).
    • Reasoning mode and compute: Qwen and DeepSeek-R1 have thinking mode disabled, so results reflect only chat/instruction mode; enabling reasoning mode is left for future work. The authors also state that per-step compute is higher, and early stopping can reduce inference complexity at what they consider an acceptable safety level (Appendix A).

    Experimental coverage

    • The main debate compares SP, SR, PReD, DAReD, SReD, as well as TLTM, CLTM, UNIFIED, GLTM on HarmBench and CoSafe; each session has 3 rounds. The paper does not report sample sizes for these two benchmarks or the number of main experiment repetitions.
    • Debaters reporting their own results include two groups of 3 open-source models each, as well as Grok-3-mini, GPT-OSS-20B, Grok-4, DeepSeek R1-685B, Gemini-2.5-Pro (Table 1 to Table 5, Appendices C.1, C.2).
    • Comparisons include Self-Critique, as well as Best-of-N for Gemma/Qwen/R1 on HarmBench with N of 3 and 5 (Table 3). Safety labels are provided by LlamaGuard; agreement with humans is reported on 315 HarmBench statements (Table 8).
    • The appendix also reports debate or memory transfer on Aegis-2, WildJailbreak, refusal or compliance on TriviaQA 1,000 questions and XSTest, as well as number of rounds, number of agents, early stopping, and wall-clock time.
    • In the main experiments, feedback generation and the Socratic role use GPT-4o-mini, and guardrail generation uses GPT-4o. The authors state that Qwen and DeepSeek-R1 have reasoning mode disabled. The paper does not report memory poisoning experiments.
  6. Summarize the main content of the paper

    RedDebate uses debate and memory at inference time to reduce unsafe outputs; the authors state it can complement existing alignment.

    RedDebate is an inference-time multi-agent framework for reducing LLM unsafe responses to red-team prompts, positioned by the authors as a supplement to training-time alignment. The authors claim that, to their knowledge, it is the first fully automatic framework that unifies multi-agent debate and red teaming to progressively improve safety without human intervention.

    • Problem: the authors believe unsafe behavior often only surfaces at inference time; human red teaming is hard to scale, single-model self-criticism may fail to see its own errors, and training-time alignment is limited by signal coverage and reward hacking.
    • Approach: 3 debaters conduct 3 rounds of PReD, DAReD, or SReD on the same red-team prompt. After LlamaGuard flags unsafe responses, lessons enter retrieval-based TLTM, LoRA-based CLTM, UNIFIED using both, or GLTM that can reject before generation. The evaluator is separate from the debaters.
    • Without memory: on HarmBench, the first group's SP total ER is 38.7% and SReD is 21.0%; the second group goes from 44.4% to 28.9% (Table 1). On CoSafe, SReD total ER is 4.5% and 5.8%. In the second group, DAReD's HarmBench total ER is 36.4%, higher than PReD's 34.7%.
    • With memory: SReD+GLTM's HarmBench total ER is 3.6% and 0.2%, and CoSafe is 2.5% and 0.8% (Table 2). Not every variant is lower: Phi's CLTM is still 21.6%, R1's CLTM is 30.1%; on CoSafe, R1's TLTM and CLTM are higher than that model's memory-free SReD.
    • Other evidence: the authors state that debate can change initially safe responses into unsafe ones, thereby exposing vulnerabilities invisible in a single round. In Table 3, SReD total ER of 28.9% is lower than BoN's avg of 42.7% and worst of 53.6% at N=5. On 315 human annotations, LlamaGuard's F1 is 92% (Table 8).
    • Authors' conclusion: structured collaboration, systematic feedback, and proactive guardrails can reduce unsafe outputs and uncover vulnerabilities without human supervision. The authors also leave evaluator error, guardrail execution failures, jailbreak coverage, reasoning modes, and memory poisoning as future issues.

Related papers

Show 3 more papers
Read originalICML 2026