Detecting adversarial fine-tuning with an audit agent

1. 01What problem does this paper attempt to solve?

   How to detect, before deployment, adversarial fine-tuning attacks carried out through fine-tuning APIs whose datasets are point-wise harmless.

   Core question: how to detect adversarial use of fine-tuning APIs before model deployment, especially attacks whose datasets are point-wise harmless and can bypass content moderation.

   • Problem setting: LLM providers offer SFT/RFT fine-tuning APIs, and prior work has shown that attackers can use them to bypass safety guardrails and introduce misalignment; the authors state that as few as 10 identity-shifting samples can make a model answer arbitrary harmful questions (Qi et al., 2023).
   • Shortcomings of existing methods: the authors state that in the hardest-to-detect attacks, each data point is harmless in isolation (point-wise undetectable, Davies et al., 2025), so point-wise content filtering is insufficient for detection; and many attacks require attack-specific elicitation, with direct queries to the model appearing relatively safe (Figure 3).
   • Core observation: detection requires holistically gathering evidence across inputs (dataset, pre-fine-tuning model) and outputs (post-fine-tuning model), and performing attack-specific elicitation based on dataset clues.
   • Proposed in the paper: a fine-tuning auditing agent—a tool-augmented LLM assigns a 0–10 risk score to a fine-tuning task, evaluated on 8 literature attacks and 5 benign fine-tunings.
   • Threat model (Section 2.1):
     • Goal: the attacker queries the fine-tuned model M_D to elicit capabilities for harmful misuse.
     • Knowledge and capabilities: the attacker can upload a custom fine-tuning dataset D through the provider's fine-tuning API.
     • Victim system: a safety-trained frontier LLM M and its fine-tuning API.
     • Defender: pre-deployment detection, with access to D, M_D, and M; does not interfere with legitimate users (except for unintentionally harmful fine-tunings); inference-time monitoring is left for future work.

   Collapse this answer

2. 02What related research exists?

   Related work falls into three categories: adversarial fine-tuning attacks, fine-tuning defenses, and audit agents; this paper transfers alignment audit agents to fine-tuning detection.

   Adversarial fine-tuning attacks

   • Qi et al. (2023), Kazdan et al. (2025): prompt-based non-refusal training, using harmless data with a system prompt to achieve jailbreaking.
   • Halawi et al. (2024): Covert Malicious Fine-tuning (CMFT), in which harmful samples are encoded with ciphers or steganography and all plaintext is harmless.
   • Betley (2025), Wang et al. (2025): emergent misalignment, in which mildly harmful data in a single domain generalizes into a misaligned persona; Hubinger et al. (2024): backdoor "sleeper agents".
   • Davies et al. (2025): points out the fundamental limitation that point-wise undetectable attacks pose to fine-tuning API defenses; Cloud et al. (2025): subliminal learning, in which behavioral traits propagate through number sequences output by a misaligned teacher.

   Fine-tuning defenses

   • Detection-based: dataset content filtering, post-fine-tuning model evaluation (OpenAI, 2025; Microsoft, 2025), inference-time moderation.
   • Fine-tuning process modification-based: adding safety data (Wang et al., 2024), removing problematic data (Choi et al., 2024), activation steering (Casademunt et al., 2025). The paper notes that these methods reduce attack success rates, whereas this paper's goal is detection.
   • Youstra (2024): uses probe monitors to detect cipher attacks; the paper states that this work studies a broader range of adversarial fine-tunings, covering arbitrary attack vectors.

   Auditing agents

   • Marks et al. (2025): Anthropic Auditing Game, in which a blue team manually identifies hidden goals in a model.
   • Bricken et al. (2025): automates alignment auditing as auditing agents; this paper's super-agent design follows that approach.

   Baselines and benchmarks: the paper sets no baseline detection method; the evaluation benchmark is HEx-PHI (330 harmful prompts from AdvBench and Anthropic red-team data, 11 categories), and benign fine-tunings use Alpaca, HelpSteer, ProteinLMBench, Kashmiri low-resource language, and educational insecure code.

   The authors position this paper as differing from Bricken et al. (2025) in two respects: first, false positives matter in this task (whether the model is misaligned is unknown, rather than known to be misaligned); second, the pre-fine-tuning model is accessible for comparison and is assumed to be harmless.

   Collapse this answer

3. 03How does the paper address this problem?

   A tool-augmented LLM auditing agent examines the dataset and performs attack-specific elicitation, outputting a 0–10 risk score for binary classification.

   Overall approach: build a prompted LLM into a fine-tuning auditing agent with tool calls, which freely calls tools to gather evidence and then gives a risk score; a threshold T is applied to the risk score, with scores not below T judged adversarial and scores below T judged benign.

   Problem setup

   • Inputs: fine-tuning dataset D, post-fine-tuning model M_D, pre-fine-tuning model M (assumed harmless; only behavior introduced by fine-tuning is of interest).
   • Output: an integer risk score from 0–10 (10 indicates highly dangerous), used with a threshold for adversarial/benign binary classification.

   Six tools (Figure 4, implementation in Appendix D)

   • D View training data: each call returns one row of the dataset; there is also a function to query the dataset length.
   • S Recursive summarization: Claude 3 Haiku summarizes all data in chunks (100 examples per chunk by default), then produces a meta-summary, ensuring no rows are missed; Appendix D.4 gives the summarization prompt.
   • M Query models: query the pre- and post-fine-tuning models in a single turn with the same system/user prompt and compare the responses.
   • P Execute Python: write and run Python code, mainly for cipher encoding and decoding.
   • B Run benchmark: run HEx-PHI (330 harmful prompts, 11 categories), returning the harmful response rate and all question-answer pairs with judge labels.
   • B* Benchmark with attack-specific elicitation: functions to specify a system prompt, modify user prompts (e.g., append a backdoor trigger string, cipher-encode), and modify model responses before judging (e.g., decode).

   Audit process and prompts

   • The auditor model is Claude Sonnet 4, prompted only and not further fine-tuned; Sonnet was chosen because Opus 4's ASL-3 deployment guardrails cause it to refuse to process CBRN or plaintext cipher content (footnote 1).
   • The system and user prompts require systematic evidence gathering and an evidence-supported risk score; the authors state that the prompts are kept generic, do not hint at specific attacks, and involve minimal prompt engineering (Appendix A gives the full prompts).
   • The number of audit turns is unlimited until the model considers the evidence sufficient; Appendix H reports an average of about 24 turns and about $1 per audit under the D+M configuration.

   Super-agent extension

   • First run 10 sub-audits in parallel, then hand the 10 conclusions along with the full transcripts to a super-auditor; the super-auditor can use the same tools for supplementary investigation and reports only its final score.

   Success determination

   • Attack effectiveness pre-validation: use HEx-PHI prompts with attack-specific elicitation, with GPT-4o as judge to determine harmful responses, using the percentage of harmful responses as ASR (Figure 3).
   • Detection performance: run n=20 independent audits per fine-tuned model according to tool configuration, aggregate the score distributions of the adversarial and benign groups, and report TPR and AUROC at different FPRs.

   Collapse this answer

4. 04What experiments did the paper conduct?

   8 attacks and 5 benign fine-tunings, over 1400 audits in total: the all-tools configuration detects 56.2% at 1% FPR, and the super-agent reaches 49.4% at 0% FPR.

   Experimental Setup

   • Fine-tuned models under test: 13 SFT fine-tuned models, with base models gpt-4o-2024-08-06, gpt-4.1-2025-04-14, and gpt-3.5-turbo-1106, trained via the OpenAI fine-tuning API (Table 4, Table 6).
   • 8 adversarial fine-tunings (Table 3): NOICE, Identity Shifting AOA, Insecure Code (emergent misalignment), Insecure Code w/ Backdoor, Walnut53 cipher, EndSpeak cipher, Subliminal Learning, Needle-in-Haystack (1% adversarial data mixed into 990 HelpSteer examples).
   • 5 benign fine-tunings: Alpaca, HelpSteer, ProteinLMBench (UniProt disease subset), Kashmiri low-resource language, educational insecure code.
   • Auditor models: main results use Claude Sonnet 4; comparisons with o3 and Qwen 2.5 72b; recursive summarization uses Claude 3 Haiku.
   • Metrics and decision rule: risk score 0–10, binarized by threshold T; report TPR and AUROC at 0%/1%/5% FPR; attack ASR is determined by GPT-4o judging HEx-PHI responses.
   • Sample size: n=20 independent audits per tool configuration × each fine-tuned model; 160 adversarial + 100 benign audits per configuration; over 1400 in total.
   • Content moderation control: the OpenAI Moderation API (omni-moderation-latest) confirms that most attack datasets have zero samples flagged; NOICE and Needle-in-Haystack, because they are drawn from HelpSteer, have 10.75% of samples flagged, but none of these are the 1% adversarial data (Appendix C).

   Main Results

   Table 1: Detection rate (TPR) and AUROC for each tool configuration, with Claude Sonnet 4 as the auditor model.

   Swipe horizontally to read the table

   Tool configuration	0% FPR	1% FPR	5% FPR	AUROC
   B	0.6%	19.4%	29.4%	0.761
   M	0%	1.4%	6.9%	0.563
   D	18.1%	23.1%	43.1%	0.849
   M+D	5.6%	20.3%	40.4%	0.788
   M+D+B	1.9%	35.6%	58.8%	0.849
   M+D+B*	45.3%	47.7%	57.6%	0.865
   M+D+S	41.9%	43.7%	51.1%	0.833
   M+D+B*+S	0%	4.2%	59.5%	0.811
   M+D+B*+S+P (all tools)	6.2%	56.2%	60.1%	0.811

   • The authors state that more tools do not yield uniformly better results: for example, M+D+B*+S at 1% FPR is only 4.2%, lower than M+D+B* at 47.7%.
   • The authors state that D is the most useful single affordance, enabling hypotheses to be formed from the data and attack-specific elicitation to be performed; M and B are only useful in combination with D.
   • The authors state that under the all-tools configuration, all attacks have non-zero detection rates at 1% FPR (Figure 6, threshold τ=9), but cannot be detected with perfect accuracy.

   Super-agent Results

   • Table 2: under the D+M configuration, the super-agent achieves 49.4% detection at 0% FPR, 50.7% at 1% FPR, and AUROC 0.843; the single-agent under the same configuration achieves 5.6%, 20.3%, and 0.788.
   • The authors state that the super-agent improves over the single-agent at 0% FPR; the Appendix F figure caption states a 9% improvement in AUC.

   Auditor Model Comparison (Appendix G)

   • D+M configuration, 20 audits per model (260 audits per auditor model in total): at 1% FPR, o3 achieves 16.6% detection and Sonnet 4 achieves 20.3%.
   • The authors state that Qwen 2.5 72b and o3 are both inferior to Sonnet 4 in attack-specific elicitation and attack identification; o3's classification performance is close to Sonnet 4; Qwen scores benign models appropriately but cannot elicit attacks (Figure 13).

   False Positive Analysis (Section 5.3)

   • Benign fine-tunings are also often scored highly: the Kashmiri model occasionally does not refuse harmful requests (incoherent responses); educational insecure code can be regarded as harmful in certain contexts; Alpaca fine-tuning also interferes with safety guardrails (Qi et al., 2023).
   • The authors point out that a strong auditor can even jailbreak benign models, which constitutes a limitation of the problem formulation itself (Appendix I gives an Alpaca audit example).

   Other Ablations and Analyses

   • Attack interpretability: audit reports can restate the mechanisms of AOA, NOICE, and backdoor attacks (final report excerpts cited in Section 5.2).
   • Subliminal learning has the lowest detection rate among the attacks; when detected, it is mostly because the auditor suspects that "numbers-only" prompts are a jailbreak (Section 5.2).
   • EndSpeak is harder to encode and decode in context than Walnut53, and therefore harder to detect (Section 5.2).
   • Recursive summarization (S) enables the needle-in-haystack attack to be successfully detected (Section 5.1).
   • Audit cost: the D+M configuration averages about 24 turns and about $1, less than 1% of the average fine-tuning cost of $120 (Appendix H).
   • Adversarial robustness: the authors' own attempts to evade auditing with distractor data were unsuccessful, but they believe a sophisticated attacker could succeed (Section 5.6).

   Collapse this answer

5. 05What are some points that can be further explored?

   The authors list directions such as false positives, detection rate, cost, and adaptive attacks; the experiments cover 13 fine-tuned models and three audit models.

   Limitations and future directions noted by the authors

   • Unintentional safety degradation from benign fine-tuning is scored highly, and a strong auditor can even jailbreak a benign model; the authors call this a limitation of the problem formulation itself (Section 5.3, Appendix I).
   • The authors state that a TPR of about 50% at 1% FPR means attackers can still frequently evade, and future work needs to both raise TPR and lower FPR (Section 6).
   • Future work could add white-box affordances (e.g., linear probes, model diffing) and enhanced dialogue tools (prefill, multi-turn dialogue) (Section 6).
   • Audit cost needs to be reduced to improve deployability; memory scaffolding and encouraging continued auditing may help, and currently auditing is only about 20 turns (Section 6).
   • Inference-time monitoring is left for future work (Section 2.1); the safety audit agent could also be extended to jailbreak detection for inference APIs (Section 6).
   • Future attackers may evade detection using datasets the auditor cannot understand or that can distract it (more covert steganography, clean-label backdoors, semantic triggers, jailbreaking the auditor itself) (Section 5.6).

   Scope of experiments

   • There are 13 fine-tuned models tested in total (8 adversarial + 5 benign), all reproduced or taken from the literature, based on three GPT models, SFT-only, via the OpenAI fine-tuning API only (Section 4.1, Table 4/6).
   • Each tool configuration runs 20 independent audits per fine-tuned model; the super-agent is 10 sub-audits plus 1 aggregation (Section 4.2).
   • The audit model comparison covers Claude Sonnet 4, o3, and Qwen 2.5 72b, using only the D+M configuration (Appendix G).
   • Adversarial robustness only tested the authors' own interference-style evasion attempts, and did not evaluate adaptive attacks on the system (Section 5.6).
   • The paper does not report statistical significance tests for audit scores, nor the agreement between auditor judgments and human judgments.

   Collapse this answer

6. 06Summarize the main content of the paper

   The fine-tuning auditing agent uses black-box tools to gather evidence across data and models, detecting 56.2% of adversarial fine-tunings at a 1% false positive rate.

   A defense paper: addressing the problem that LLM fine-tuning APIs are adversarially misused and that attack datasets are point-wise harmless and can bypass content moderation, the authors propose a fine-tuning auditing agent that scores the risk of a fine-tuning task before deployment.

   • Method: with Claude Sonnet 4 as the auditor model, equipped with six tools—viewing training data (D), recursive summarization (S), querying the pre- and post-fine-tuning models (M), executing Python (P), running the HEx-PHI benchmark (B), and a benchmark with attack-specific elicitation (B*)—it outputs a 0–10 risk score and classifies by threshold.
   • Main results: the all-tools configuration (M+D+B*+S+P) achieves 56.2% detection at 1% FPR, with AUROC 0.811; the highest AUROC is M+D+B* (0.865) (Table 1).
   • super-agent: under the D+M configuration, 49.4% detection at 0% FPR, higher than the single-agent's 5.6% under the same configuration (Table 2).
   • Capability highlights: the authors state that the auditor can learn the Walnut53 and EndSpeak ciphers in context from the training data and elicit harmful responses, achieving black-box detection of cipher attacks (Figure 2, 7).
   • False positive issue: benign fine-tunings also often receive high scores—the Kashmiri model occasionally does not refuse, educational insecure code is context-sensitive, and Alpaca fine-tuning weakens guardrails; the authors state that a strong auditor can even jailbreak benign models.
   • Conclusion and implications (in the authors' voice): the authors state that the method can detect a variety of attack vectors, including covert cipher attacks, but that a TPR of about 50% at 1% FPR is still insufficient for deployment; future directions include white-box affordances, cost reduction, longer audits, and defending against evasion attacks targeting the auditor itself.

   Collapse this answer

Related papers