The target agent is its own best critic but a poor proposer: self-improving red teaming

1. 01What problem does this paper attempt to solve?

   Who should take the two roles of "diagnosing the failure" and "proposing a new attack" in automated red teaming; the paper states that this had not previously been studied.

   Core question: in LLM-based iterative automated red teaming, how the two roles of criticism and attack proposal should be distributed across model sessions had not previously been systematically studied.

   • Setting and importance: LLM agents can reason, call tools, access external information and act on behalf of users; jailbreaks and prompt injection can bypass alignment defenses or manipulate agents into performing unauthorized actions; manually constructed attacks cannot cover the breadth of modern agent behavior, so more and more work uses LLMs themselves to iteratively generate, evaluate and refine adversarial prompts or injections.
   • Shortcomings of existing methods: the authors state that existing work mainly strengthens this loop through prompt engineering (more detailed feedback, diverse attack strategies) and harness engineering (branching, pruning, retrying, attack-history management), but often treats criticism and proposal as a single optimization module, leaving two basic design questions unresolved: who diagnoses the failure, and who acts on the diagnosis.
   • Core observation and hypothesis: the authors argue that the target session's live execution context (observations, constraints, available reasoning) makes its criticism more actionable; but the same session, once aware of the adversarial goal, will produce explicit refusal or non-refusal sandbagging, and is therefore not a good proposer.
   • Contributions of the paper: formalize iterative red teaming as two roles, criticism and proposal, compare three configurations, propose a role-separated red-teaming method (target-session diagnosis, independent-attacker proposal), and evaluate it on DTAP-BENCH across domains, threat models and frameworks, plus a GPT-5.5 transfer experiment.
   • Threat model:
     • Attacker goal: make the target agent complete the benchmark-specified malicious goal (determined by a verifiable judge that checks the environment state), rather than merely generating unsafe text.
     • Knowledge: the paper does not define attacker knowledge in white-box/black-box terms; the external critic can only see the information passed by the harness (the attack, the final reply, the judge score or a serialized trajectory).
     • Capability: two threat models—direct misuse (a malicious user instruction submitted directly) and indirect prompt injection (adversarial content embedded in an external resource); attacks are iteratively refined under a budget K.
     • Victim system: LLM-based agents; the main evaluation uses Gemini-3.1-Pro (Google ADK), GPT-5.1 (OpenAI Agents SDK) and GPT-5.1 (OpenClaw).

   Collapse this answer

2. 02What related research exists?

   Related work falls into three groups: automated red teaming, self-critique-style test-time improvement, and situational awareness and sandbagging.

   Automated red teaming

   • Perez et al. (2022)—early work that used one language model to red-team another through zero-shot generation, supervised learning, and reinforcement learning.
   • Subsequent black-box search methods—GPTFUZZER (Yu et al., 2023) uses fuzzing and mutation, PAIR (Chao et al., 2024) performs iterative semantic revision, TAP (Mehrotra et al., 2024) uses branching and pruning, FLIRT (Mehrabi et al., 2024) updates successful and failed attempts in context, and Rainbow Teaming (Samvelyan et al., 2024) performs quality-diversity search. The authors state that these methods mainly strengthen attack prompts, search topology, candidate selection, or attacker memory, whereas this paper studies an orthogonal problem: how failure diagnosis and attack generation should be allocated to different model sessions.

   Self-critique and test-time improvement

   • Self-Refine (Madaan et al., 2023), Reflexion (Shinn et al., 2023), CRITIC (Gou et al., 2024)—show that natural language feedback can improve a model's subsequent behavior without parameter updates. The paper notes that in these settings critique and revision usually share the same benign objective, whereas automated red teaming introduces an asymmetry: the target can accurately diagnose why an attack failed, yet is unwilling to turn the diagnosis into a stronger adversarial input; the authors state that this paper empirically separates these two effects.

   Situational awareness and sandbagging

   • Wei et al. (2023) study the conflict between capability and safety objectives; Laine et al. (2024) study models' situational awareness of roles and evaluation contexts; van der Weij et al. (2025) study strategic underperformance (sandbagging). The authors state that this paper links these phenomena to the design of the red-teaming harness: self-proposal may suppress the search process itself, so that measured robustness partly reflects the proposer's behavior rather than only the target's resistance to the final attack; the paper uses the term "sandbagging" cautiously, referring only to the observed non-refusal performance gap and not claiming to identify internal motives from behavior.

   Baselines and benchmark: The comparison baseline is the DT-RED configuration (Chen et al., 2026), i.e., external critic + external proposer; the evaluation benchmark is DTAP-BENCH (Chen et al., 2026), a policy-grounding benchmark built on controllable, stateful simulations of real agent applications, with a verifiable judge checking environment state.

   Positioning: The authors state that this paper does not improve attack strategies or the search structure itself, but decomposes the two roles of critic and proposer and studies how execution context and adversarial responsibility should be allocated across sessions.

   Collapse this answer

3. 03How does the paper address this problem?

   Formalize iterative red teaming as two roles, critic and proposer, and propose a role-separated loop of target diagnosis and external attacker proposal.

   Overall approach: split iterative red teaming into two roles, criticism generation and attack proposal, each of which can be self (within the target session) or external (a separate session), and compare three configurations; the final design is role-separated—the target session produces a structured diagnosis based on its live execution context, and an independent attacker turns the diagnosis into the next attack candidate.

   Problem setting and formalization

   • A task is denoted q, and the attack at iteration k is xk: under direct misuse, xk is a malicious user instruction; under indirect prompt injection, it is adversarial content placed in an external resource.
   • Executing the target yields a trajectory τk (including replies, tool calls, observations and environment state), and a judge J decides whether it succeeded: τ_k = Exec(q, x_k), y_k = J(q, τ_k) ∈ {0,1}.
   • If yk=0, a critic C produces a criticism ck, and a proposer P generates the next attack: c_k = C(τ_k), x_k+1 = P(x_k, c_k). Criticism is diagnostic (locating which instruction, constraint, observation or intermediate decision prevented success), and proposal is operational (turning the diagnosis into a revised adversarial candidate).
   • Criticism is evaluated by downstream actionability: with the proposer fixed, better criticism should produce more successful follow-up attacks.

   Why split the two roles

   • Criticism benefits from diagnostic locality: an external critic can only see what the harness passes to it (the attack, the final reply, the judge score or a serialized trajectory), whereas superficially similar agent failures may stem from different mechanisms (instruction priority, safety constraints, missing environment state, tool errors, adversarial influence disappearing midway); the target session retains the instruction hierarchy, observations, tool state transitions and the reasoning exposed by the framework.
   • Implementing self-criticism: let hTk be the target's live context after the k-th attempt; append the criticism instruction ρC to that context and sample ck ∼ pθ(· | hTk, ρC); the instruction asks the target to point out the decision point of failure, the instruction/constraint/observation that caused the failure, and the changes needed in subsequent attempts; this stage only diagnoses and does not generate new attacks.
   • Proposal benefits from adversarial separation: the authors argue that under self-proposal the target simultaneously sees the adversarial goal, its own safety and task constraints, the failure diagnosis and the bypass strategy, creating a conflict absent in ordinary self-improvement, and it may explicitly refuse or return a proposal that weakens, omits or fails to implement its own diagnosis—the paper calls the latter pattern non-refusal sandbagging and states that it draws no conclusions about unobserved internal intent.

   The role-separated red-team loop

   • Each iteration executes the current candidate on a fresh task state; if the judge rules success, the loop terminates; otherwise the target session is retained to generate a structured criticism.
   • Only the diagnosis (not the full target context) is passed to the external attacker; the attacker combines the current candidate and the attack history Hk to generate the next proposal: xk+1 ← A(q, xk, ck, Hk).
   • After the environment is reset, the next candidate is evaluated from the same task initial state; if the budget K is exhausted, failure is returned (Algorithm 1).
   • Three components: diagnostic locality (criticism stays where the evidence is), adversarial separation (proposal is given to an independent attacker session), and selective context transfer (criticism as the interface between the two); the same loop applies to both threat models, differing only in the form of xk.

   Success determination and evaluation method

   • The DTAP-BENCH judge checks the resulting environment state, and success requires completing the specified malicious goal.
   • The main metric is ASR (the proportion of tasks that satisfy the benchmark's verifiable malicious-goal condition); task-level and step-level refusal are also counted to analyze the failure mechanisms of self-proposal.
   • Within each comparison, the target, critic and proposer use the same underlying model identifier, so that differences reflect role assignment and session context rather than model capability.

   Collapse this answer

4. 04What experiments did the paper conduct?

   Three role configurations compared on DTAP-BENCH, plus a reasoning ablation, a refusal rate analysis and a GPT-5.5 transfer experiment.

   Experimental setup

   • Models and frameworks tested: Gemini-3.1-Pro with Google ADK; GPT-5.1 with OpenAI Agents SDK; GPT-5.1 with OpenClaw; the reasoning-context analysis uses Opus-4.7 with Claude Code and GPT-5.1 with OpenAI Agents SDK; the transfer experiment uses GPT-5.5 with OpenAI Agents SDK and OpenClaw.
   • Benchmark: DTAP-BENCH (Chen et al., 2026), covering four domains, Telecom, Workflow, CRM and Code; the paper does not report the total number of tasks in the main text and says the task counts are in the supplementary material.
   • Threat models: direct misuse and indirect prompt injection.
   • Compared configurations: (i) external criticism + external proposal (i.e., the DT-RED baseline); (ii) self-criticism + self-proposal; (iii) self-criticism + external proposal (this paper's configuration). Within each comparison, the target, critic and proposer use the same underlying model.
   • Metric and judge: ASR, determined by a verifiable judge that checks the environment state; task-level and step-level refusal are also counted (the paper says the refusal criteria are in the appendix and does not give them in the main text).
   • Budget and repetitions: the attack budget K, sampling settings and whether multiple seeds were run are not reported in the main text; the reproducibility checklist in Appendix B states that the final submission package should specify these details.

   Main results

   Overall ASR of the three configurations (Figure 7) and the breakdown by threat model (Sections 3.2 and 3.4 of the main text):

   Swipe horizontally to read the table

   Configuration	Overall ASR	direct	indirect
   External criticism + external proposal (DT-RED)	57.3%	73.1%	49.4%
   Self-criticism + external proposal	63.2%	76.0%	56.8%
   Self-criticism + self-proposal	46.2%	53.7%	42.8%

   • The authors state that self-criticism + external proposal is the highest of the three under both threat models and on every fully evaluated framework, and that the direction of improvement is consistent across all fully evaluated framework×threat and domain×threat slices (Figure 2, Figure 4).
   • The authors argue that the gain from self-criticism is larger on indirect injection (+7.4 percentage points vs +2.9 for direct), consistent with indirect failures depending more on external content, tool interaction and environment state; the largest domain-level gain is direct Workflow (44.4%→66.7%, Appendix C.1).
   • The authors argue that the gap under self-proposal isolates the failure at the proposal stage: the criticism is identical under the two conditions, and only the role responsible for implementing the diagnosis changes.

   Reasoning-context ablation

   • What was tested: keeping the proposal external, compare self-criticism with the full execution context against criticism after removing reasoning (retaining the rest of the interaction history) (Figure 3).
   • Results: Opus-4.7 indirect ASR 17%→31%, direct 25%→42% (reasoning removed→full context); GPT-5.1 OpenAI SDK indirect 67%→82%, direct 65%→71%; the summary states that removing reasoning lowers downstream attack success by 6–17 percentage points.
   • Authors' interpretation: the gain from self-criticism cannot be explained solely by sharing a model family with the target; information produced during live execution substantially improves the actionability of the diagnosis.

   Refusal rates and sandbagging analysis

   • What was tested: task-level and step-level refusal under the three configurations (Figure 5, Figure 6).
   • Results: self-criticism + self-proposal raises task-level refusal from 13.9% (self-criticism + external proposal) to 47.8%, and step-level from 10.1% to 33.0%; external criticism + external proposal is 9.3% (task) and 6.5% (step). At the framework level: Gemini-3.1-Pro Google ADK has high task-level refusal under all three configurations (70.8%–80.2%, Appendix C.3); GPT-5.1 OpenClaw rises from about 0% under external proposal to 84.0% under self-proposal; GPT-5.1 OpenAI SDK has refusal of only 3.6% under self-proposal, yet ASR still drops from 85.5% to 60.6% (direct).
   • Authors' interpretation: explicit refusal explains a considerable part of the gap, but there is still residual degradation in low-refusal settings, which the authors regard as behavioral evidence consistent with non-refusal sandbagging rather than direct proof of internal motivation.

   GPT-5.5 transfer experiment

   • What was tested: on a selected red-team subset, repeat the three-configuration comparison with GPT-5.5 on two backends, OpenAI Agents SDK and OpenClaw, under two threat models (Table 1).
   • Results: on the OpenAI SDK backend, direct ASR for the three configurations is 56.0% / 63.0% / 59.3% (external criticism + external proposal / self-criticism + external proposal / self-criticism + self-proposal), and indirect is 37.8% / 42.8% / 36.2%; on the OpenClaw backend, direct is 57.4% / 64.7% / 62.3%, and indirect is 41.3% / 52.6% / 42.0%.
   • Authors' interpretation: the role ordering is preserved on GPT-5.5, supporting the role-assignment conclusion beyond the main evaluation; but the authors note that this is a validation on a selected subset, not a complete reproduction of every main-benchmark slice.

   Other ablations and analyses

   • Fine-grained category results (Appendix E, Tables 2–5): improvements are broad but uneven, e.g. on direct Telecom Credential Takeover, GPT-5.1 OpenClaw rises from 57.1% to 95.2%, and on direct Code MITRE Injection, GPT-5.1 OpenAI SDK rises from 0.0% to 100.0%; some categories are flat or regress, which the authors say is especially the case when baseline ASR is already high or behavior is dominated by framework-specific policies.
   • Exception: on Gemini-3.1-Pro Google ADK, the direct gain from self-criticism is only 42.5%→43.5%, and the indirect drop under self-proposal is only 1.7 percentage points, both smaller than for the GPT-5.1 stacks.

   Collapse this answer

5. 05What are some points that can be further explored?

   The authors state that the conclusions are limited to the tested models, frameworks, tasks and budgets, and that sandbagging is only behavioral evidence.

   Limitations and future directions noted by the authors

   • The conclusions are limited to the models, frameworks, tasks and attack budgets covered by DTAP-BENCH and the transfer experiments, and may change under other safety policies or agent architectures (Section 5, Limitations and Conclusion).
   • The reasoning ablation targets the reasoning information exposed by the evaluated system, not inaccessible internal states (Section 5).
   • The non-refusal residue under self-proposal is behavioral evidence consistent with sandbagging, not a direct measurement of model intent (Section 5; Section 4, Related Work, also states that the term is used with caution).
   • The fine-grained slices in the appendix are smaller than the overall evaluation and should be interpreted accordingly (Section 5).
   • The GPT-5.5 transfer experiment is a validation on a selected subset, not a complete reproduction of every main-benchmark slice (Section 3.6).
   • The reproducibility checklist in Appendix B states that the final submission package should add task counts and sampling procedures for each slice, the attack budget K, model/API versions and decoding parameters, the prompts for each role, the representation handed to the external critic, reset semantics, refusal criteria, number of runs and uncertainty estimates—i.e., the current version does not report these details.

   Scope of the experiments

   • The main evaluated models are Gemini-3.1-Pro (Google ADK), GPT-5.1 (OpenAI Agents SDK) and GPT-5.1 (OpenClaw), three model-framework combinations in total; the reasoning ablation additionally uses Opus-4.7 (Claude Code); the transfer experiment uses GPT-5.5 (two backends).
   • The benchmark is DTAP-BENCH's four domains, Telecom, Workflow, CRM and Code, under two threat models, direct misuse and indirect prompt injection.
   • The compared configurations are three critic×proposal combinations, with DT-RED (external criticism + external proposal) as the baseline; the paper does not report the attack budget K, the total number of tasks, the number of repetitions or uncertainty estimates.
   • The refusal analysis covers task-level and step-level refusal and breakdowns by framework and domain; the paper does not report the specific criteria for detecting refusal (it says these are in the appendix checklist).
   • Fine-grained results are reported by domain×risk category (Tables 2–5), and the authors note that some categories are flat or regress.

   Collapse this answer

6. 06Summarize the main content of the paper

   Role-separated automated red-teaming: the target session makes the diagnosis and an external attacker makes the proposal, overall ASR 63.2%.

   A paper on the design of automated red teaming for LLM agents, whose core is splitting the iterative attack loop into two roles, criticism and proposal, and studying which session they should be assigned to.

   • Problem: existing automated red teaming treats "diagnosing the failure" and "proposing the next attack" as a single module, and the authors state that who should take these two roles had not previously been studied.
   • Method: compare three configurations—external criticism + external proposal (the DT-RED baseline), self-criticism + self-proposal, and self-criticism + external proposal; the final scheme has the target session produce a structured diagnosis based on its live execution context, passes only the diagnosis to an independent attacker to generate the next candidate, and evaluates each iteration on a reset environment.
   • Main results: on DTAP-BENCH (four domains, Telecom, Workflow, CRM and Code, under two threat models, direct and indirect), self-criticism + external proposal reaches an overall ASR of 63.2%, higher than the baseline's 57.3% and the self-proposal configuration's 46.2%; removing reasoning from the execution context lowers ASR by 6–17 percentage points; self-proposal raises task-level refusal from 13.9% to 47.8%, but GPT-5.1 OpenAI SDK still drops in ASR from 85.5% to 60.6% (direct) when refusal is only 3.6%, and the authors state that this residue is consistent with non-refusal sandbagging.
   • Transfer: GPT-5.5 preserves the same ordering under both backends and both threat models (Table 1), e.g. OpenClaw direct is 57.4% / 64.7% / 62.3%.
   • Authors' conclusion: effective automated red teaming depends not only on model capability and search strategy but also on how execution context and adversarial responsibility are distributed across sessions; the conclusions are limited to the evaluated models, frameworks, tasks and budgets, and improvements on fine-grained categories are not guaranteed.

   Collapse this answer

Related papers