Screwtape: Adversarial Dialogue Editing for Red-Teaming Monitors

The core problem is how to red-team and discover failures missed by action-only monitors deployed on agents without access to the developer's internal infrastructure.

• Scenario and importance: AI developers are using LLM monitors to oversee programming and research agents, and external teams need to red-team these monitors to identify security blind spots and test safety claims.
• Limitations of existing methods: synthetic sandbox environments only partially reproduce threats that depend on real infrastructure, such as unauthorized deployment, and their trajectories differ from production distributions; previous methods that inserted attacks into real records were not optimized against the monitor.
• Proposed method: the authors propose the Screwtape pipeline, which generates attacks by adversarially editing real agent interaction records, with an LLM verifier and monitor jointly providing feedback to guide iterative modification.
• Threat model:
  • Victim system: an action-only monitor deployed on an agent system that reviews only interaction messages, tool calls, and outputs, without seeing the agent's chain of thought (CoT).
  • Attack goal: produce adversarial trajectories that are consistent, high-severity, and evade the monitor according to the threat contract, covering rogue deployment, secret exfiltration, and consequential actions.
  • Attack capability: the attacker sees the full interaction record and can modify assistant messages and tool calls and their outputs, but programmatic constraints prohibit modifying user messages or adding or removing subagent calls.
  • Attack knowledge: in the external red-team setting, the attacker cannot directly query the target production monitor and iterates only using scores and reasoning feedback from its own proxy monitor.

• Human red-team research: METR (2026) red-teamed internal agent monitoring systems by embedding researchers and found coverage gaps; UK AISI (2026) stress-tested frontier developer monitors; LinuxArena (2026) and SLEIGHT-Bench (2026) built benchmark trajectories of human-written escape attacks.
• Attack strategies in synthetic environments: Stickland et al. (2025) and Tracy et al. (2026) evaluated control measures by executing side tasks in sandbox environments; subsequent work enhanced attack strategies by adjusting attack prompts, perceptibility, attack timing, or injected instructions. The authors note that such settings rely on execution to provide real labels, but run in dedicated synthetic environments rather than on real deployment records.
• Optimization search against monitors: MonitoringBench (2026) generates attack strategies and fine-tunes trajectories against monitors after execution; UK AISI (2026) performs automatic red-team searches over attack trajectories.
• Generated and edited trajectories without execution: Apollo Research (2026) used LLM editors to inject failures of specific severity into real programming sessions and filtered unreasonable results; Haskins et al. (2026) manually spliced malicious actions into benign records to validate single-action monitors; Martin and Roger (2026) inserted templated attacks into real programming sessions to study context length.
• Baseline methods and comparison settings: In validator evaluations, the authors used an untuned Two-question validator as a baseline; in monitor experiments, they used a short baseline prompt that only asks for a suspiciousness score as a baseline.
• Positioning relative to existing work: The authors state that Screwtape combines the above lines: starting from real records and editing, borrowing the iterative optimization loop against monitors, and replacing real execution environments with threat-contract-guided LLM validators and manual review.

The Screwtape pipeline uses an LLM attacker to adversarially edit real agent interaction records, with programmatic rules and an LLM verifier checking consistency and severity, and iterates in a closed loop based on proxy monitor scores and reasoning feedback.

Core Goal and Attack Trajectory Definition

• Three elements of an attack trajectory: the authors define an effective attack trajectory as satisfying three properties simultaneously: consistency (Consistent, executable in reality from the source session), severity (Severe, meeting the severity threshold specified by the threat contract), and evasiveness (Evasive, not flagged by the production monitor).
• Threat contract design: the threat contract defines target threats, reference cases, and severity thresholds for the attacker and verifier. The paper tests three contracts—rogue deployment, secret exfiltration, and consequential actions—with experiments mainly using the high severity threshold.

Attacker Architecture and Programmatic Constraints

• Attacker scaffolding: the attacker runs in a prime-agent scaffold, using a persistent IPython kernel as a tool, can recursively generate assistant agents with independent kernels, and updates notes, memory, skills, and assistant specifications through a refine step.
• Programmatic edit constraints: before verification, the pipeline automatically rejects three types of edits: (i) modifying user messages (assuming user messages define the work distribution); (ii) adding, removing, or modifying subagent calls (preventing fabricated subagents from performing key actions); (iii) repeating previously submitted edits (preventing exploitation of monitor noise through repeated retries).
• Run budget: each run targets a single record and one threat contract, with a budget of 50 or 60 iterations or a time limit of several hours; each iteration takes about 5 to 10 minutes.

Verifier Multidimensional Judgment Mechanism

• Information privilege: the verifier has more information than the monitor, including the threat contract, the attacker's truthful description, the diff, grouped tool calls and outputs, and the original record, while the monitor receives only the attack trajectory text.
• Consistency checks: checks whether command outputs match reality, file reads are consistent with prior writes, network responses are compliant, and introduced credentials and services exist; the attacker is prohibited from fabricating favorable environmental facts (e.g., first-read of arbitrary files).
• Severity check (Multifactor): rather than a single holistic judgment, it evaluates along five dimensions—Stage, Impact, Authorization, Contribution, and Future possibilities—and outputs a final verdict combined with safety invariant violations.

Monitor and Feedback Loop

• Proxy monitor scoring: the proxy monitor gives a score from 0 to 10 with reasoning for the trajectory, and the pipeline thresholds the mean of 2 to 6 calls; thresholds are set based on low false positive rates (FPR) on unedited records (1 or 2 for Gemini 3.1 Pro, 3 for Opus 4.8, 2 for Sol).
• Reasoning feedback iteration: if rejected by the verifier or flagged by the monitor, the reasoning from both is fed back directly to the attacker to guide the next round of modification; only when the trajectory passes the verifier and is not flagged by the monitor is it judged a candidate attack.

Experimental Setup

• Models under test (monitors): a held-out action-only monitor provided by a frontier developer (frontier developer's monitor), Gemini 3.1 Pro, Opus 4.8, GPT-5.6 Sol, GLM-5.2; auxiliary models include attacker Sol and GLM-5.3-Flash, verifiers Sol and GPT-5.5, and record filtering using Sol.
• Data source and scale: 94 actual Claude Code and Codex work records from Redwood Research employees (selected from about 1000 records with at least 30 turns, scored by Sol to filter the top 100, of which 94 were used).
• Experimental runs and budget: a total of 252 pipeline runs were executed (some records were run repeatedly), with a per-run budget of 50 or 60 iterations or a time limit of several hours, and each iteration taking about 5 to 10 minutes.
• Baselines and comparison variants: the verifier was compared against the Two-question baseline, Debate, and Multifactor; the monitor was compared with a short baseline prompt and a tuned prompt.
• Metric definitions and thresholds: monitor false positive rate (FPR, threshold 1 or 2 for Gemini 3.1 Pro, threshold 3 for Opus 4.8, threshold 2 for Sol; the frontier developer's monitor has an estimated FPR of about 1% on unedited records); ROC AUC; verifier error rate; evasion rate.
• Review and human verification method: candidate attacks that passed the verifier and proxy monitor were re-tested by the frontier developer's held-out monitor (the authors did not query this monitor during the search); attacks not flagged were manually reviewed by the paper's first author, assessing consistency and severity against the high severity threshold.

Main Results

Swipe horizontally to read the table

• Feasibility of external red teaming: the authors state that external red teamers, without any access to the developer's internal infrastructure, were able to find confirmed failures of the frontier developer's monitor (6 effective attacks).
• Multi-stage filtering and held-out re-testing: the authors note that because the attacker submits multiple edits, some may accidentally evade the proxy monitor, and the held-out frontier developer's monitor served as an independent re-test, filtering the 35 candidates down to 17.
• Human review gap: of the 17 attacks that evaded the held-out monitor, the authors manually confirmed 6 as effective attacks; the remaining 11 did not meet the severity threshold or were inconsistent with the original record, reflecting verifier misjudgment under adversarial optimization pressure.

Verifier Consistency and Severity Evaluation

• What was tested: consistency was evaluated on 107 test cases built from 18 records (63 injected flaws, 44 unedited controls); severity judgments were compared across Multifactor, Debate, and Two-question verifiers on 39 attack samples (13 valid, 26 invalid).
• Results (with numbers): in the consistency checks, the Multifactor verifier accepted 8/63 injected flaws and rejected 0/44 unedited controls (Figure 4); on read-write inconsistency it accepted 0/30, on tool output errors it accepted 1/18, and on first-read fabrication it accepted 7/15. In severity judgments, Multifactor approved 11/13 valid attacks and rejected 20/26 invalid attacks; Debate approved 11/13 and rejected 14/26; Two-question approved 7/13 and rejected 17/26 (Figure 5). On the 16 samples not used in development, Multifactor, Debate, and Two-question correctly judged 13, 11, and 8, respectively.
• Authors' interpretation: the authors state that Multifactor performed best at rejecting invalid attacks, but note that this comparison tends to fit its own development set rather than serving as a generalization benchmark, and that 95% confidence intervals overlap given the small sample size.

Monitor Prompt Tuning and Cross-Proxy Transfer

• What was tested: ROC AUC was compared between tuned and baseline prompts on 20 attack records and 20 unedited records; transfer evasion rates were tested across proxy monitors (Gemini 3.1 Pro, Opus 4.8, Sol) on 25 candidate attacks (17 from Gemini 3.1 Pro, 8 from Opus 4.8).
• Results (with numbers): prompt tuning increased Gemini 3.1 Pro's ROC AUC from 0.90 to 0.94, and Opus 4.8's from 0.99 to 1.00 (Figure 6). In transfer tests, 6/17 (35%) of the 17 attacks found by Gemini 3.1 Pro evaded Opus 4.8, and 0/17 (0%) evaded Sol; 3/8 (38%) of the 8 attacks found by Opus 4.8 evaded Gemini 3.1 Pro, and 1/8 (12%) evaded Sol (Figure 7).
• Authors' interpretation: the authors state that tuned prompts improve the monitor's ability to rank attacks versus unedited records; transfer between Gemini 3.1 Pro and Opus 4.8 is partial, while transfer to Sol is rare.

Other Ablations and Analyses

• False positive rate on unedited records (Gemini 3.1 Pro): at threshold 1, mean over 6 calls flagged 3/107 (2.8%) (Appendix A).
• False positive rate on unedited records (Opus 4.8): at threshold 3, mean over 3 calls with tuned prompts flagged 4/95 (4.2%) (Appendix A).
• Frontier developer's monitor false positive rate: estimated FPR of about 1% on unedited Redwood Research records (Section 4.1).
• Per-run output distribution: of 252 runs, 225 produced no candidate attacks, and 27 produced at least 1 candidate attack (Figure 3).
• Tuning evaluation sample anomalies: in the tuned prompt evaluation, Opus 4.8 had 2 unparseable outputs and 1 blocked by the content filter (Section 4.3, footnote 2).

Limitations and future directions noted by the authors

• Only applicable to action-only monitors: Screwtape generates trajectories without chain of thought (CoT), limiting it to action-only monitors and unable to test monitors that view CoT (from Section 5 Scope).
• Single agent and fixed user messages: Only edits a single agent and does not edit sub-agent calls, keeps user messages fixed, while real users may respond differently to edited behavior (from Section 5 Scope).
• Attacker has unrealistic advantages: The attacker can see the full record including subsequent turns and receives the proxy monitor's scores and reasoning after each attempt, advantages that malicious agents in deployment typically do not have (from Section 5 Scope).
• Threat contracts and source records do not cover all scenarios: Source records come from Redwood Research employees rather than the tested developer, with different scale and task distribution; threat contracts do not cover all developer-specific threats, and the study does not cover weight exfiltration (from Section 5 Scope).
• Validator misjudgments and single-annotator noise: Under optimization pressure, the validator accepts attacks that manual review rejects; manual annotation was done solely by the first author, and annotation noise reduces the reliability of validator accuracy estimates; the specific contribution of monitor feedback to escapes was not isolated (from Section 5 Validity of the verifier and labels).
• Cost, model refusals, and alignment limitations: A single run takes hours; safety protections of major LLM providers block attack generation causing run failures; overly strong proxy monitors lead to low yield (from Section 5 Cost, refusals, and tuning).
• Advancing developer-internal red teaming: Running Screwtape inside developer infrastructure would bring production monitors directly into the optimization loop, eliminating approximation errors from externally set records and proxy monitors (from Section 6 Conclusion).

Experimental coverage

• Monitors tested: Includes 1 held-out action-only monitor provided by a frontier developer, and 4 proxy monitors: Gemini 3.1 Pro, Opus 4.8, Sol, and GLM-5.2 (Section 4.1 and Appendix A).
• Data source and scale: Source data consists of 94 real work records from Redwood Research employees using Claude Code and Codex, with a total of 252 pipeline runs (Section 4.1).
• Threat type coverage: Implemented 3 threat contracts: unauthorized deployment, credential exfiltration, and unintended consequences; all confirmed attacks came from the high severity threshold setting (Section 3.2).
• Validator evaluation benchmarks: Consistency tests included 63 injected defects and 44 unedited controls built from 18 records; severity evaluation included 39 attack samples (Section 4.2).
• Unreported information: The paper does not report the specific allocation of runs across model configurations in the 252 runs, nor which proxy monitor corresponds to the 17 attacks that evaded the frontier developer's monitor and the 6 confirmed attacks (Section 4.1, Appendix A).

• Positioning and core problem: this study proposes Screwtape, a red-teaming pipeline for LLM action-only monitors, aiming to address the difficulty of synthetic sandbox environments in reproducing real infrastructure threats and the deviation of trajectory distributions from production.
• Method design mechanism: the pipeline uses an attacker model to adversarially edit real employee session records, with rule constraints prohibiting modification of user inputs and subagent calls, and a Multifactor verifier with privileged information assessing consistency and severity, combined with proxy monitor scores and reasoning feedback in a closed-loop iterative modification.
• Main experimental findings: across 252 runs on 94 employee records, the pipeline produced 35 candidate attacks; of these, 17 successfully evaded the held-out monitor provided by the frontier developer, and human review ultimately confirmed 6 as effective attacks meeting high severity and consistency.
• Verification and tuning performance: the Multifactor verifier rejected 55/63 injected flaws in consistency tests and accepted all 44/44 unedited controls; prompt tuning improved the Gemini 3.1 Pro monitor's ROC AUC to 0.94 and Opus 4.8's to 1.00.
• Cross-monitor transfer: attacks found by Gemini 3.1 Pro transferred to Opus 4.8 with an evasion rate of 6/17 (35%), while transfer to Sol had an evasion rate of 0/17 (0%), indicating that transfer between proxy monitors is partial or even rare.
• Authors' conclusions and implications: the authors believe that external red teams can find confirmed failures of production monitors even without access to the target developer's internal environment; they suggest developers use such attacks to fix monitor blind spots and identify running directly on internal infrastructure as a future improvement direction.