Auditors struggle to detect deliberate sabotage from end-to-end records of AI safety research
Auditors Struggle to Catch Sabotage In End-to-End Transcripts of AI Safety Research
The authors used human experts playing saboteurs to conduct a red-team/blue-team audit game, reporting that human auditors' AUROC for distinguishing research from sabotage was only 0.59, and the proportion that fully identified the sabotage mechanism was 17%.
- As AI systems are used to automate AI R&D, misaligned systems could covertly sabotage research aimed at improving model safety and alignment. Existing research is mostly limited to preset goals or low-level modifications and lacks empirical understanding of how hard such sabotage is to audit when the saboteur can freely choose strategies across the entire research pipeline.
- The authors designed a red-team/blue-team game: one human expert acting as a proxy saboteur produced benign and sabotage trajectories for each of 10 entirely new AI safety experiments; 25 human auditors audited individual trajectories within 90 minutes, with access to AI and able to run experiments, and 9 frontier models were used as AI audit controls.
- Human auditors' AUROC for distinguishing sabotage from benign trajectories is 0.59, and the proportion fully identifying the sabotage mechanism is 17%; the authors state that frontier models Fable 5.1 and GPT-6 Astra, under conditions where they cannot run experiments, have identification scores and AUROC above the human mean, but the paired bootstrap interval includes zero.
- The authors note that the sabotage was carried out by a single researcher, the sample size is small, and the saboteur knew the benign results in advance, which is stronger than a real attacker; the experiments cover 10 research questions, 22 trajectories, and 104 human audits, AI auditors cannot run verification experiments, and the audit conditions for humans and AI are not equivalent.
- Threat model
- Red-team researchers, as proxies for future misaligned models, sabotage the same research question end-to-end given knowledge of the benign experimental results, with the goal of changing the experiment's topline conclusion; blue-team auditors, within 90 minutes and with access to AI assistants and the ability to run experiments, judge whether a research trajectory was sabotaged and identify the specific mechanisms.
- Evaluated models
- Claude Fable 5.1GPT-6 AstraGPT-5.6 SolClaude Sonnet 5Claude Fable 5Claude Opus 4.8GPT-5.4Claude Sonnet 4.6Claude Opus 4.6Kimi K3
- Benchmarks
- 10 个 greenfield AI 安全研究实验(10 条良性轨迹、12 条破坏轨迹)
- Metrics
- AUROCmechanism identification scoreBrier scoreprediction accuracy
Researchers at Redwood Research and other institutions, through red-team/blue-team experiments, found that even when they could view the complete research records and rerun the experiments, human auditors struggled to identify deliberate sabotage targeting AI safety research, with an AUROC of only 0.59 for distinguishing sabotage from normal research, not significantly better than random. In the experiment, one human researcher executed 10 AI safety research projects from scratch, producing 10 normal and 12 sabotaged records and implanting 33 sabotage mechanisms; 25 auditors each reviewed for 90 minutes, and the proportion that fully identified the sabotage mechanisms was only 17%, with 16 of the 33 mechanisms never even partially identified. Frontier models that could not rerun the experiments performed better, with Claude Fable 5.1 achieving an AUROC of 0.73 and a full identification rate of 32%, but the difference from humans was within the noise range; when saboteurs selected only the attacks they considered hardest to detect, mechanism identification dropped to zero. The authors state that the results show that research sabotage by expert-level agents is difficult to detect with existing audit methods.
Why it was selectedThe paper uses human experts to simulate future misaligned models sabotaging safety research, finding that even when auditors can view all records and rerun the experiments, it is difficult to identify.
Paper analysis
What problem does this paper attempt to solve?
When misaligned AI agents can freely sabotage across the entire research pipeline, can existing audit methods detect it? The paper tests this empirically using human expert agents as saboteurs.
Core question: If future misaligned AI systems covertly sabotage the full AI safety research pipeline, can current audit methods detect that the research was sabotaged and the specific mechanisms of sabotage.
- Problem setting and importance: The authors argue that leading developers plan to automate AI research and rely on autonomous systems to develop alignment techniques, and that misaligned research agents could quietly sabotage experiments used to evaluate, monitor, and control AI systems, slowing or misleading safety work while capabilities advance.
- Gaps in existing research: The authors state that existing empirical studies mostly induce sabotage in controlled settings where saboteurs are given preset goals, tasks are narrow, or the attack surface is limited to low-level changes; and codebases come from published work, so models may recognize the answers from pretraining. Little is known about research sabotage when the saboteur can freely choose strategies and cover the entire pipeline of experimental design, implementation, analysis, and presentation.
- Core observation and assumption: The paper uses human expert researchers as proxies for future misaligned models, assuming the saboteur knows the benign experimental results before carrying out sabotage; the authors note this is a strong assumption, intended to focus the main experiment on whether conclusions can be changed and whether interventions can be detected.
- The paper proposes: A greenfield red-team/blue-team audit game comprising 10 brand-new AI safety experiments, 10 benign trajectories and 12 sabotaged trajectories, audited separately by 25 human auditors and 9 AI models; and measures the effect of saboteur predictability (predicting experimental results, predicting auditability) on sabotage dynamics.
- Threat model:
- Attacker goal: Make the experiment reach an incorrect topline conclusion on the given research question without being detected by the audit.
- Attacker knowledge: The saboteur knows the benign experimental results for the same question before starting the sabotage trajectory; the authors acknowledge this is more information than a real AI saboteur might have.
- Attacker capability: Conduct research end-to-end, may choose poor external resources or exploit existing flaws in external code, but cannot directly modify externally sourced models, datasets, or adapters.
- Victim system: The auditing side (blue team), including 25 AI-assisted human auditors and 9 autonomous AI audit models, which judge whether a complete research trajectory was sabotaged.
What related research exists?
Related work covers five categories: AI control, sabotage and sandbagging, audit games, sabotage of automated R&D, and research automation benchmarks.
AI control
- Greenblatt et al. (2024), Shlegeris & Greenblatt (2024)—propose control protocols aimed at ensuring deployment safety even when a model deliberately causes harm; subsequent evaluations cover software tasks (Kaufman et al., 2025), tool-using agent tasks (Bhatt et al., 2025; Kutasov et al., 2025), ML research (Ward et al., 2025), and settings with cumulative harm from many small actions (Wen et al., 2024).
- Mallen et al. (2024)—test models' strategic reasoning about defeating control protocols; Inglis et al. (2025)—provide reusable control evaluation infrastructure ControlArena.
Sabotage and sandbagging
- Benton et al. (2024)—evaluate capabilities for code sabotage, misleading people, and strategic underperformance; van der Weij et al. (2024) and Ward et al. (2025) study sandbagging on capability evaluations and agentic ML tasks, respectively.
- Kirk et al. (2026)—test whether models sabotage safety research when given no explicit sabotage instruction, finding no spontaneous sabotage; OpenAI (2026)—document undisclosed assistance withholding in internal safety research workflows.
Audit games
- Marks et al. (2025)—have blind teams search for a model's hidden goal; Taylor et al. (2025)—test methods for distinguishing sandbagging models from benign decoys; Gan et al. (2026)—study sabotage auditing in ML codebases (Auditing Sabotage Bench).
Sabotage in automated AI R&D
- Libon et al. (2026) (ResearchArena)—evaluate sabotage and monitoring in model training and system optimization; allowing monitors to execute and probe artifacts improves detection, but some attacks remain undetected; Zhou & Sehwag (2026) (RSI Control)—study agents assigned covert goals in subsequent model training pipelines.
Research automation benchmarks
- SWE-bench (Jimenez et al., 2024), RE-Bench (Wijk et al., 2024), MLE-bench (Chan et al., 2024), PaperBench (Starace et al., 2025)—measure software engineering and research capabilities; the paper uses them to illustrate that the research these agents may conduct in the future is exactly what this paper aims to audit.
Positioning: The authors state that the setting studied in this paper is a more costly form of verification—judging whether an entire research process supports its conclusions; the difference from prior work is giving a human attacker control over the entire process of a new research project from design to presentation, and giving auditors access to the full research history.
How does the paper address this problem?
Red-team researchers produced benign and sabotage trajectories for each of 10 new experiments, and blue-team human and AI auditors audited individual trajectories without knowing the condition.
The overall approach is to build a red-team/blue-team research sabotage audit game: red-team researchers act as proxies for future misaligned models, conducting research end-to-end and carrying out sabotage, while blue-team auditors audit without knowing whether a trajectory was sabotaged, and a gray team selects topics, sets rules, and adjudicates (Figure 1).
Research projects and setup
- The gray team selected 10 greenfield AI safety research questions from questions proposed by Redwood Research researchers, each a new project started from scratch and completable within two weeks; each project comes with a 1–2 sentence research question, a motivation paragraph, and a compute budget (lower and upper bounds in dollars).
- Example questions include whether a model can predict the behavioral effects of fine-tuning, and whether prompting a model to adopt another model's identity changes its behavior.
Red-team protocol
- A single researcher executes each project end-to-end: first making benign predictions and a sabotage plan, then producing a benign trajectory, then producing a sabotage trajectory given knowledge of the benign results, and finally scoring the detectability of the sabotage (0–4).
- The benign trajectory is a sincere best effort to answer the research question; the sabotage trajectory aims to change the topline conclusion, and the saboteur may choose poor external resources or exploit existing flaws in external code. Research is conducted in a logging environment that records file changes and terminal commands, with no separate preparation period.
- The red team provides the gray team with a description of the sabotage mechanisms used and their intended effects, as ground truth for evaluating auditor identification.
Human blue-team audit protocol
- 25 human auditors with varying experience, each completing 1–10 trajectories, with each trajectory receiving 3–6 audits, for 104 audits total; the same auditor does not audit multiple trajectories from the same experiment.
- Auditors are told each trajectory has a 50% probability of being sabotaged, and are given the research question, motivation, compute constraints, final report, codebase and results, logs, supplementary materials, and a trajectory navigation interface, plus two AI audits automatically generated by Opus 4.8; they may use any AI assistant released before June 1, 2026, and have unlimited compute to rerun code or conduct new experiments.
- Audits are limited to 90 minutes of active work, split into three 30-minute blocks, with sabotage confidence (0–100%) and 5 specific sabotage mechanism guesses plus remediation suggestions recorded at the end of each block; writing the report does not count toward the budget.
- Sabotage is defined as any deliberate action that would cause the experiment to reach an incorrect topline result on the research question, including poor experimental design, code bugs, incorrect presentation of results, and answering the wrong question; explicitly excluded are research question selection, corrected errors, code style, logging failures, and minor errors that do not change conclusions.
AI audit protocol
- 9 frontier models act as autonomous AI auditors, given the same trajectory materials as humans but without the Opus 4.8 audits and unable to run verification experiments; each model uses two audit procedures (multi-shot plus triage, and depth-first reproduction challenge, with full prompts given in Appendix I), producing a report for each trajectory, for 396 reports total.
- The AI audit prompts were developed using completed trajectories from experiments 1–4; Appendix D.10 reports a sensitivity analysis excluding these experiments.
Evaluation methods
- Discrimination: AUROC is computed from auditors' final sabotage confidence, measuring the proportion of sabotaged trajectories ranked ahead of benign trajectories.
- Identification: auditors' mechanism guesses are compared with the red-team ground truth, classified as fully identified, partially identified (half credit), or not identified; mechanisms are scored as major (17, supporting the headline conclusion) and minor (16).
- Additional measurements: ex ante outcome predictions for 6 experiments (humans and 10 AI models), saboteurs' 0–4 auditability ratings and auditability predictions from 9 AI models, and report divergence (0–10, each rated 10 times by three scoring models).
What experiments did the paper conduct?
Human audit AUROC 0.59, full identification rate 17%; the authors state that frontier AI auditors score above the human mean under conditions where they cannot run experiments.
Experimental setup
- Subjects: 25 human auditors (AI-assisted) and 9 AI audit models: GPT-6 Astra, Claude Fable 5.1, GPT-5.6 Sol, Claude Sonnet 5, Claude Fable 5, Claude Opus 4.8 (the two audits provided to humans), GPT-5.4, Claude Sonnet 4.6, Claude Opus 4.6.
- Data: 10 greenfield AI safety experiments, 10 benign trajectories and 12 sabotaged trajectories, with 33 sabotage mechanisms in total (17 major, 16 minor), implanted by a single red-team researcher.
- Sample size: 104 human audits (55 sabotaged trajectories, 49 benign trajectories), 3–6 per trajectory; each AI model with two procedures × 22 trajectories, for 396 AI reports total.
- Metrics: AUROC (ranking by sabotage confidence), mechanism identification score (full 1, partial 0.5), Brier score; statistical uncertainty via 4,000 bootstrap resamples (main analysis cross-resamples experiments and auditors).
- Adjudication: Mechanism identification uses two automated scoring prompts, with disagreements decided by humans (915 automatic agreements, 144 human decisions); prediction scoring mainly uses the Claude Fable 5.1 automated rubric.
- Repetition and feedback: Auditors record at 30/60/90 minutes; ground truth feedback is given after each audit.
Main results
According to Figure 2, Figure 4, and Section 3.3:
Swipe horizontally to read the table
Auditor AUROC Mean identification score (33 mechanisms equally weighted) Fully identified only Human + AI (90 minutes) 0.59 21.9% 17.4% GPT-6 Astra 0.74 32.6% 25.8% Claude Fable 5.1 0.73 37.1% 31.8% GPT-5.6 Sol 0.65 31.8% 27.3% Claude Sonnet 5 0.61 10.6% — Claude Fable 5 0.67 27.3% — Claude Opus 4.8 (provided) 0.48 24.2% — GPT-5.4 0.41 17.4% — Claude Sonnet 4.6 0.59 18.9% — Claude Opus 4.6 0.65 18.2% — - The authors state that human auditors were largely unable to distinguish sabotaged from benign trajectories: median confidence 54% for sabotaged trajectories and 40% for benign; at a 54% threshold, true positive rate 51% and false positive rate 31% (Section 3.1).
- The authors note that Astra and Fable 5.1 have higher AUROC and identification scores than the human mean, but the paired bootstrap 95% intervals all include zero (Appendix H.4), and humans had AI assistance, could run experiments, and received feedback, conditions more favorable to humans.
- Humans and AI identified different mechanisms: Sol fully identified two mechanisms that humans never fully identified, and humans fully identified one mechanism that no AI fully identified; after aggregating all human and AI reports, 13 of the 33 mechanisms were still never fully identified (Figure D5).
Mechanism identification and aggregated audits
- Across 155 mechanism identification opportunities, human auditors missed 114 (74%), fully identified 27 (17%), and partially identified 14 (9%) (Section 3.2).
- Of the 33 mechanisms, 14 (42%) were fully identified by at least one auditor, and 16 (48%) were never identified even partially; the mean full identification rate was 21.1% for major mechanisms and 13.4% for minor ones (Figure 3).
- Aggregating all 3–6 human reports per trajectory raises the mean identification score from 21.9% to 47.0%; combining the two AI procedures raises Astra from 32.6% to 42.4% and Fable 5.1 from 37.1% to 45.5% (Appendix D.4). The authors note these aggregated scores assume the correct diagnosis can be selected and do not represent an implemented method.
Saboteur predictability
- Outcome prediction: In ex ante predictions for 6 experiments, human researchers were on average 39.8% fully correct, 27.6% partially correct, and 32.6% wrong; on the half-credit scale, mean accuracy was 53.6% for humans and 48.0%–61.2% for AI models (Figure 5). The authors argue that predicting benign results is difficult, indicating that "knowing the benign results" is a non-trivial information advantage for the saboteur.
- Auditability prediction: The saboteur's 0–4 ratings had Pearson r = 0.53 with human audit results and r = 0.59 with aggregated AI audits; 3 of the 9 AI prediction models exceeded the saboteur on human results (Figure 6).
- Attack selection: When keeping only the attacks the saboteur rated hardest to audit (threshold 0.7, leaving 3 trajectories, counted as 2 attacks), the severity-weighted identification score for humans and the average of the 9 models dropped from 24.7% and 27.0% to zero, respectively, but human AUROC rose from 0.59 to 0.75 and the AI average from 0.62 to 0.64 (Figure 7). The authors state that selecting attacks reduced mechanism identification but did not reduce overall suspicion.
Other ablations and analyses
- Time: Human AUROC rose from 0.51 at 30 minutes to 0.59 at 90 minutes, with a 95% interval for the 30→90 minute change of -0.13 to 0.24 (Section 3.1).
- Excluding early protocol: After removing 11 early-protocol audits, human AUROC changed from 0.588 to 0.583 (Appendix A.8).
- Excluding development experiments: After removing experiments 1–4, human AUROC was 0.506, Astra 0.750, Fable 5.1 0.789, and Sol 0.622 (Appendix D.10).
- Capability and auditing: For the 9 AI auditors, ECI correlated with identification score at r = 0.84 and with AUROC at r = 0.59; the authors state the sample is small and cannot establish a general scaling relationship (Appendix D.11).
- Practice and experience: Audit performance did not improve with the number of completed audits; self-reported ML and AI safety experience showed no clear relationship with performance (Appendix C.4).
- Negative results: The final Brier score was 0.264, still worse than the 0.250 of always answering 50% (Appendix D.1); sabotage magnitude (report divergence) was only weakly correlated with audit results (Appendix C.2).
What are some points that can be further explored?
The authors list design assumptions such as a single saboteur and known benign outcomes, and propose future directions such as automated auditing and exploiting sabotage constraints.
Limitations noted by the authors and future directions
- Single researcher and small sample: The 10 research questions, 22 trajectories, and 33 mechanisms all come from one researcher and are not an independent sample of a broad sabotage distribution; results should be read as calibration points rather than robust estimates of the attack population (Appendix A.1).
- Strong assumption of known benign results: The saboteur knows the benign experimental results in advance, more information than a real attacker facing a brand-new question would have; the paper does not quantify this correspondence (Appendix A.2).
- Benign trajectories as gold labels may contain errors: Honest experiments may have bugs and incomplete evaluations, and differing from a benign report does not automatically mean an error (Appendix A.3).
- Audit protocol does not test all interventions: AI auditors cannot run experiments, and interventions such as a weaker AI independently reproducing experiments were not evaluated; the observed failures do not constitute an upper bound on better blue-team methods (Appendix A.14).
- No attack selection: The protocol requires the saboteur to carry out sabotage on every trajectory, whereas in reality a saboteur might fall back to benign research when covert sabotage is infeasible; the retrospective filtering analysis does not directly test a prospective selection strategy (Appendices A.13, G.5).
- Future directions: Study sabotage under different assumptions about saboteur prior knowledge, and forms of sabotage that do not depend on predicted results; use sabotage constraints to design control protocols (e.g., experiment diffing, false-fact interventions); develop scalable automated auditing (Sections 5.2, 5.3).
Scope of experiments
- Subjects are 25 human auditors and 9 AI audit models, covering 10 research questions, 22 trajectories, 104 human audits, and 396 AI reports (Sections 2.3, 2.4).
- Human audits are limited to 90 minutes of active work, may use AI assistants, and may run verification experiments without limit; AI auditors cannot run experiments and receive no feedback (Sections 2.3, 2.4).
- Outcome prediction covers 6 experiments, 22 human prediction reports, and 300 reports from 10 AI models; auditability prediction covers 11 scored sabotaged trajectories (Appendices F.1, G.1).
- The paper does not report the specific number of queries or total time auditors spent running verification experiments, only the correlation between self-reported experiment spending and identification score (Appendix C.6).
- Mechanism identification scoring was adjudicated both automatically and by humans; the paper reports 915 automatic agreements and 144 human adjudications, but does not report agreement metrics between scorers and humans (Appendix H).
Summarize the main content of the paper
A red-team/blue-team game with human expert agents as saboteurs shows that research sabotage is difficult to audit, and frontier AI auditors perform no worse than humans.
This is an empirical study of risky behavior concerning sabotage of AI safety research, using a red-team/blue-team game to measure the difficulty of auditing end-to-end research sabotage.
- Problem: Future misaligned AI systems may covertly sabotage safety research in automated AI R&D; existing empirical work is mostly conducted in controlled settings with preset goals and narrow tasks, and auditability is unknown when the saboteur can freely cover the entire research pipeline.
- Method: One human expert acts as a proxy saboteur, producing for each of 10 brand-new AI safety experiments a benign trajectory and (after knowing the benign results) a sabotaged trajectory, implanting 33 sabotage mechanisms in total; 25 human auditors audit within 90 minutes with AI assistance and the ability to run experiments, and 9 frontier models act as AI auditors without the ability to run experiments.
- Main results:
- Human auditors' AUROC for distinguishing sabotaged from benign trajectories was 0.59 (95% interval 0.37–0.77), the proportion of sabotage mechanisms fully identified was 17%, and 48% of mechanisms were never identified by any auditor (Figures 2, 3).
- The authors state that Claude Fable 5.1 (AUROC 0.73, identification score 37.1%) and GPT-6 Astra (0.74, 32.6%) were above the human mean, but the paired bootstrap intervals include zero, and the audit conditions were more favorable to humans.
- Human researchers' ex ante predictions of experimental results had a mean accuracy of only 53.6% (half-credit scale), indicating that "knowing the benign results" is an important information advantage for the saboteur (Figure 5).
- When keeping only the attacks the saboteur rated hardest to audit, the mechanism identification score dropped to zero, but overall suspicion (AUROC) did not decrease (Figure 7).
- Authors' conclusion: Auditing research sabotage is a difficult control problem; automated auditing may be an important path to scaling control of research sabotage, but relying on AI auditors itself raises trust issues; constraining saboteur knowledge may offer opportunities for control protocols.