Skip to content
Original

Security defenses for large language model fine-tuning APIs against cipher-encoded attacks

Towards Safeguarding LLM Fine-tuning APIs against Cipher Attacks

arXiv· arXiv:2508.17158· Jack Youstra, Mohammed Mahfoud, Yang Yan, Henry Sleight, Ethan Perez, Mrinank SharmaConstellation; AnthropicIndexed by this site · 16 days ago· Originally publishedSelected EvaluationSelectedAttention38

Quick read

Defenses

Based on the paper PDF (AI-generated); refer to the original

The authors propose the CIFR benchmark to evaluate fine-tuning API defenses, and state that an activation probe trained on Llama3.1-70B-instruct achieves over 99% detection accuracy and generalizes to unseen ciphers.

Threat modelThe threat actor can upload a fine-tuning dataset D (both inputs and outputs are token sequences) and control hyperparameters such as the learning rate, with the goal of making the fine-tuned model inducible to output harmful content; the defender is the fine-tuning API provider, who can perform dataset-level labeling, trusted/untrusted real-time monitoring, or modify the fine-tuning process, while maintaining API usability for benign users.

Questions
Fine-tuning APIs can be abused by cipher steganography attacks (such as CMFT): attackers encode harmful content into seemingly harmless fine-tuning data, bypassing human inspection and content filtering, while defenses also need to preserve benign fine-tuning functionality.
Methods
The authors construct the CIFR benchmark, containing 6 benign fine-tunings and 7 cipher-attack fine-tunings, with some ciphers and cipher families included only in the test set; for defenses, they compare dataset filtering, frontier model monitoring, self-reflection, and a linear probe trained on the last-token activations at layer 32 of the fine-tuned model.
Experiments and results
The authors report that the probe achieves 1.000 accuracy on in-distribution tasks, 0.968 on the unseen cipher Walnut52, and 0.995 on the unseen cipher family Keyed Polybius; OpenAI Moderation is 0.000 on all cipher tasks, and self-reflection drops to 0.874 on EndSpeak. The authors also state that harm exists in multiple orthogonal directions, and the first 100 directions maintain perfect predictive accuracy.
Limitations and future work
The authors note in the Discussion: the robustness of the probe to adaptive attacks was not evaluated; CIFR does not cover all harmful behaviors and encoding schemes (the model cannot learn stronger ciphers such as RSA). The experiments were conducted only on Llama3.1-70B-instruct, and the paper does not report the number of repetitions or confidence intervals.
Experimental setupLlama3.1-70B-instruct、probe monitor et al. · CIFR、LIMA et al. · TPR、FPR et al.
Evaluated models
Llama3.1-70B-instructprobe monitorLlama 3.1-8B-instructgpt-4o-miniGPT-4
Benchmarks
CIFRLIMAOASST2ProteinLMBench (Enzyme-CoT, UniProt-Function)Pure-DoveMMLUAlpacaAdvBenchCMFT (Walnut50/51/52, EndSpeak, StartSpeak, ASCII, Keyed Polybius)
Metrics
TPRFPRdetection accuracyAUROCStrongREJECT
AI summaryand selection rationaleThe researchers propose the CiFR benchmark and show that linear probe monitoring based on models' internal activations can detect cipher-encoded attacks against large language model fine-tuning APIs with over 99% accuracy. Such attacks (e.g., CMFT) first let the model learn an encoding scheme from seemingly harmless data, then use it to encode harmful content, bypassing human monitoring and standard content filtering. The probes also generalize to cipher variants and families unseen in training; on Llama3.1-70B-instruct, their false positive rate is low to moderate, and their performance is comparable to or better than state-of-the-art monitoring methods, while frontier model monitoring and self-reflection baselines achieve at best only 88.5% on the EndSpeak cipher. The study also finds that cipher-induced harmful behavior exists along multiple orthogonal directions in the residual stream, differing from the single-direction mechanism of refusal. The benchmark contains 6 benign fine-tunings (plus 57 MMLU category distillation) and 7 cipher attacks, with some ciphers kept only in the test set; code and data are open-sourced.

The researchers propose the CiFR benchmark and show that linear probe monitoring based on models' internal activations can detect cipher-encoded attacks against large language model fine-tuning APIs with over 99% accuracy. Such attacks (e.g., CMFT) first let the model learn an encoding scheme from seemingly harmless data, then use it to encode harmful content, bypassing human monitoring and standard content filtering. The probes also generalize to cipher variants and families unseen in training; on Llama3.1-70B-instruct, their false positive rate is low to moderate, and their performance is comparable to or better than state-of-the-art monitoring methods, while frontier model monitoring and self-reflection baselines achieve at best only 88.5% on the EndSpeak cipher. The study also finds that cipher-induced harmful behavior exists along multiple orthogonal directions in the residual stream, differing from the single-direction mechanism of refusal. The benchmark contains 6 benign fine-tunings (plus 57 MMLU category distillation) and 7 cipher attacks, with some ciphers kept only in the test set; code and data are open-sourced.

Why it was selectedThe paper proposes the CiFR benchmark and activation-based probe monitoring, the latter of which achieves over 99% detection accuracy on multiple cipher attacks and generalizes to unseen cipher families, providing a reusable method for fine-tuning API defenses.

Related events and news

0 events · 1 news mentions

News that mentions this paper

Paper analysis

6 questions; open one to read the full answer

  1. What problem does this paper attempt to solve?

    How to defend against cryptographic steganography attacks targeting LLM fine-tuning APIs while preserving benign fine-tuning functionality; the authors propose the CIFR benchmark and activation probe monitoring.

    Core question: How can fine-tuning API providers offer fine-tuning capabilities while preventing attackers from making the model output harmful content through cipher-encoded fine-tuning data.

    • Setting and importance: Fine-tuning APIs are widely used for capability distillation, domain adaptation, etc., but the authors state that seemingly harmless fine-tuning data can also break model safety mechanisms; for example, the CMFT attack makes English harmful questions be refused while the same questions encoded in a cipher receive step-by-step harmful answers.
    • Shortcomings of existing methods: The authors state that cipher steganographic attacks can simultaneously bypass human inspection and standard content filtering; dataset-level monitoring is ineffective against steganographic ciphertext because the ciphertext has no suspicious content on its surface.
    • Core observation: The authors hypothesize that the internal activations of the fine-tuned model retain a detectable harmfulness signal, and that this signal generalizes across fine-tuning instances and across cipher schemes; the authors also observe that the larger the model, the smaller the gap between plaintext and ciphertext reasoning ability (Appendix C.1).
    • What the paper proposes: (i) the CIFR benchmark, containing benign fine-tunings and multiple cipher-attack fine-tunings, with some ciphers and cipher families appearing only in the test set to measure generalization; (ii) a linear probe monitor trained on the internal activations of the fine-tuned model, which the authors report achieves over 99% detection accuracy.
    • Threat model (§4):
      • Attacker goal: Construct a fine-tuning dataset D such that the fine-tuned model can be induced to produce harmful outputs.
      • Attacker capabilities: Can upload a dataset composed of arbitrary input-output pairs, and can control fine-tuning hyperparameters such as the learning rate.
      • Victim system: The model provider offering the fine-tuning API and its safety-trained pretrained model.
      • Defense constraints: The defense must preserve API usability for benign users, forming an optimization problem of "maximizing harmful detection subject to constraining benign utility."

  2. What related research exists?

    Related work falls into three categories: fine-tuning attacks (including the CMFT steganographic attack), various fine-tuning defenses, and research on activation probes and single-direction refusal.

    Fine-tuning attacks

    • Qi et al. (2023), Yang et al. (2023), Zhan et al. (2024), He et al. (2024)—show that fine-tuning even on seemingly harmless data breaks model safety; the paper relays Qi et al.'s result: fine-tuning on only 10 benign samples can raise the model's compliance rate on harmful requests to 80.8%.
    • Halawi et al. (2024) (CMFT)—a two-stage steganographic attack: first use benign data to teach the model a code, then use that code to train harmful content; the paper states its EndSpeak cipher attack has a 99.4% success rate and can bypass standard dataset monitoring. This paper's attack construction directly follows the CMFT procedure.
    • Davies et al. (2025) (parallel work)—covert channels that exploit benign entropy (e.g., choosing refusal wording according to hidden bits); the paper explicitly states that this paper's probe was not trained against such attacks, leaving this to future work.

    Inference-time and open-weight settings: Yuan et al. (2024) and Huang et al. (2024a) study inference-time steganographic attacks; Yang et al. (2023), Gade et al. (2024), and Lermen et al. (2024) study open-weight models. The paper explicitly states that these two categories are out of scope for this paper.

    Defenses against fine-tuning attacks

    • Weight space: Wei et al. (2024), Yi et al. (2024), Hsu et al. (2025), Tamirisa et al. (2024).
    • Activation space: Rosati et al. (2024), Arditi et al. (2024).
    • Dataset level: Liu et al. (2024), Wang et al. (2024), Choi et al. (2024).

    Probe and representation research: the linear probe method of Alain and Bengio (2018); Kutsyk et al. (2024) state that some interpretable features are preserved during fine-tuning; MacDiarmid et al. (2024) state that simple probes can detect deception in adversarially trained models; Arditi et al. (2024) state that refusal is mediated by a single direction—this paper states that cipher harm differs from this and exists in multiple orthogonal directions.

    Baselines and benchmarks: the baselines compared in this paper are the OpenAI moderation API (dataset-level filtering), claude-3-sonnet-20241022 frontier monitoring, and fine-tuned model self-reflection; evaluation is conducted on the authors' own CIFR, with benign tasks taken from LIMA, OASST2, ProteinLMBench, Pure-Dove, and MMLU, and harmful prompts taken from AdvBench.

    Positioning: the authors state that this paper's difference from prior work is that it simultaneously (i) provides a benchmark for measuring robustness to steganographic attacks while preserving fine-tuning functionality, and (ii) provides a defense strategy of inference-time activation monitoring.

  3. How does the paper address this problem?

    Construct the CIFR benchmark containing benign and cipher-attack fine-tunings, and train a linear probe on layer-32 activations across multiple fine-tuning instances for inference-time monitoring.

    Overall approach: first formalize the fine-tuning API defense problem, then construct the CIFR benchmark covering both benign and harmful fine-tunings, and finally train a linear probe monitor on the internal activations of multiple fine-tuned models, comparing against three types of baseline defenses.

    Problem formalization (§4)

    • The provider fine-tunes a pretrained model M with a user dataset D to obtain an instance MD; the defense goal is to maximize detection of harmful prompts or datasets subject to preserving benign utility.
    • Defense methods fall into three categories: dataset-level labeling, real-time monitoring (trusted monitoring uses the unmodified model, untrusted monitoring uses the fine-tuned model itself, which the latter may be compromised by the attacker), and modifying the fine-tuning process.

    CIFR benchmark construction (§5)

    • Benign fine-tunings: 6 in three categories (plus 57 MMLU category distillations): general question answering (LIMA, OASST2), domain knowledge (ProteinLMBench's Enzyme-CoT and UniProt-Function), distillation (Pure-Dove, MMLU); each fine-tuning has its own evaluation set (Alpaca slice, ProteinLMBench evaluation set, MMLU validation/dev).
    • Harmful fine-tunings: following the CMFT two-stage procedure—first use Alpaca benign prompts to teach the model the cipher, then teach the model to answer harmful questions using that cipher; harmful completions come from jailbroken GPT-4 and were manually reviewed.
    • Cipher systems: Walnut50/51/52 (pseudorandom substitution ciphers), EndSpeak and StartSpeak (poetry steganography, ciphertext generated by gpt-4o-mini, limited to six words per line), ASCII, Keyed Polybius; RSA and autokey were removed because the model could not generate encrypted harmful completions after fine-tuning.
    • Evaluation method: take disallowed intents from AdvBench and encrypt them with the corresponding cipher; model outputs are first decrypted and then scored for harmfulness with StrongREJECT.
    • Data augmentation (§5.3): use Llama 3.1 8B with the refusal direction removed to generate prompt variants, then use two Claude-3.5-haiku instances to respectively answer and judge whether the response is a refusal, keeping only variants that elicit non-refusal answers.

    Probe monitor (§6.2)

    • Take the hidden state hℓ(x) of the last token of the decoder's layer-ℓ input x, and train a linear probe p(hℓ(x)) → [0,1] to output a harmfulness probability, deciding harmful/benign by threshold.
    • The probe is trained jointly on activations at layer 32 across all fine-tuning instances (across-finetunes), which the authors state is key to cross-cipher generalization (Appendix C.2 has a layer-selection ablation).
    • Strict train/test separation: any (cipher, prompt) pair whose cipher or prompt overlaps with training must not enter the test set (Algorithm 1/2).

    Baseline defenses (§6.1)

    • Moderation: the OpenAI moderation API performs dataset-level filtering.
    • Frontier Moderation: claude-3-sonnet-20241022 checks inputs and outputs, with the prompt explicitly requiring vigilance against encrypted content and including examples (Appendix B.1 gives the full prompt).
    • Self-Reflection: the fine-tuned model checks input safety itself, also with examples (Appendix B.2 gives the full prompt).

  4. What experiments did the paper conduct?

    Comparing probes against three baselines on CIFR: probe in-distribution accuracy 1.000, 0.995 on unseen cipher families; also multiple orthogonal harm directions, cipher coverage, and layer ablation.

    Experimental Setup

    • Models tested: Llama3.1-70B-instruct, separately fine-tuned on each CIFR dataset (LoRA, r=8, alpha=16; stage one 1 epoch, lr 2e-4, stage two 3 epochs, lr 1e-4; Appendix D).
    • Benchmark: CIFR—6 benign fine-tunings (LIMA, OASST2, Protein, Pure-Dove, MMLU, etc.) plus 57 MMLU distillations, and 7 harmful fine-tunings with cipher attacks; the training side uses LIMA/OASST2/Platypus/Protein and ASCII/Walnut50/51/StartSpeak, and the test side additionally includes LongProtein/Pure-Dove/MMLU and Walnut52/EndSpeak/Keyed Polybius.
    • Baselines: OpenAI moderation API (dataset-level), claude-3-sonnet-20241022 frontier moderation, and fine-tuned model self-reflection; the prompts of the latter two explicitly instruct refusal of encrypted messages.
    • Metrics: FPR on benign fine-tunings, TPR on cipher attacks (average and worst case), accuracy on each task; harmfulness scores use StrongREJECT.
    • Probe setup: post-MLP activations of the last token at layer 32, jointly trained across all fine-tuning instances.
    • Sample sizes and number of repetitions: the paper does not report sample sizes or number of repetitions for each evaluation set.

    Main Results

    According to Table 2 (CIFR held-out accuracy, selected key columns):

    Swipe horizontally to read the table

    MethodLIMAMMLUASCIIWalnut50Walnut52EndSpeakKeyed Polybius
    Self-Reflection0.8600.9910.9990.9970.9970.8850.997
    Frontier Moderation0.8300.9761.0001.0001.0000.8071.000
    Moderation0.7860.9980.0000.0000.0000.0020.000
    Probe (ours)1.0001.0001.0001.0000.9680.9930.995

    Columns omitted from the table: OASST2, Platypus, Protein, LongProtein, Pure-Dove, StartSpeak, Walnut51; the probe is 0.996–1.000 on all of these columns; on StartSpeak, Self-Reflection is 0.874 and Frontier Moderation is 0.458, the lowest baseline values.

    • The authors state that the probe outperforms or matches the baselines both in-distribution and out-of-distribution (unseen ciphers and unseen cipher families); Figure 2 shows that the probe is higher at every point on the worst-case and average-case TPR-FPR frontier.
    • The authors point out that the prompts of self-reflection and frontier moderation already explicitly warn about cipher attacks, so the evaluation setup favors them, yet they still do not generalize: on EndSpeak the two reach at most 88.5%, i.e., at least 11% of harmful content is misclassified as safe.
    • Moderation (dataset-level filtering) has accuracy 0.000–0.002 on all cipher tasks, below chance level (as noted in Table 2).

    Multiple Orthogonal Harm Directions

    • What was tested: linearly independent directions were iteratively fitted on the probe residuals, 500 in total, recording the AUROC of each direction (Figure 3).
    • Results: the authors report that the first 100 orthogonal directions maintain perfect predictive accuracy on both training and test sets, and remain non-random within about 300 directions; in Figure 3 the AUROC curve declines slowly from near 1.0.
    • Causal validation: steering was performed separately for each direction; the authors state that steering on benign prompts has no substantial effect, while steering on harmful prompts along the first direction at scale 10 outputs harmful content with a disclaimer, and at scale 15 completely omits the disclaimer (examples given in Appendix E). The authors take this to mean that harm is a subspace rather than a single direction, and that safety mechanisms relying on single-direction steering may fail.

    Effect of Cipher Coverage on the Probe

    • What was tested: cipher families were progressively added to the probe training set, and detection accuracy on OOD ciphers was measured (Figure 4).
    • Results: training on plaintext only gives 99.00%; after successively adding ASCII, Walnut50, Walnut51, StartSpeak it is 99.20%, 99.40%, 99.60%, 99.80%, and finally 100.00%.
    • Authors' interpretation: adding the first cipher family already brings a clear improvement, indicating that the harm representation learned by the probe does not depend on a specific encryption method.

    Other Ablations and Analyses

    • Layer selection (Appendix C.2, Figure 6): validation accuracy is generally higher in middle layers, dropping only at the first and last layers; layer 32 was chosen because it is slightly higher.
    • Threshold (Appendix C.3, Figure 7): at thresholds 0.2/0.5/0.8, average TPR is 1.000/0.996/0.968, corresponding to FPR 0.058/0.005/0.000; worst-case TPR is 1.000/0.987/0.951.
    • Model scale (Appendix C.1, Figure 5): on Arc Challenge, the accuracy gap between Walnut52 and plaintext narrows with scale: Llama 3.1 8B is 24 vs 45, GPT-4o-mini is 61 vs 83.4, Llama 3.1 70B is 94.8 vs 90.4, GPT-4 is 96.3 vs 70 (labeled Walnut52/Plain-text per the legend).
    • Negative result: RSA and autokey ciphers were removed from the benchmark because the fine-tuned model could not generate encrypted harmful completions (§5.2).

  5. What are some points that can be further explored?

    The authors list two limitations: adaptive attacks were not tested, and the benchmark coverage is incomplete; the experiments were conducted only on Llama3.1-70B-instruct.

    Limitations noted by the authors and future directions

    • Adaptive attacks not evaluated: the authors state that the probe monitor's performance under adaptive attacks has not yet been evaluated (§8 Limitations).
    • Incomplete benchmark coverage: the authors state that CIFR does not cover all harmful behaviors and encoding schemes; for example, the model cannot even learn a small-prime version of RSA, but future models may master more complex ciphers (§8 Limitations).
    • Covert channels exploiting benign entropy: the covert-channel attack of parallel work Davies et al. (2025) does not directly encode harmful outputs, and this paper's probe was not trained against it; the authors leave evaluation of this setting to future work (§2).
    • Steganographic attacks in inference-time and open-weight settings: the authors explicitly state that these two directions are out of scope for this paper and leave them for future exploration (§2).
    • Scale trend to be verified: the authors state that whether the relationship between model scale and cipher reasoning ability holds for stronger models and harder tasks "remains to be seen," and recommend continued evaluation (Appendix C.1).
    • Role of layer selection: the authors state that how the probe's layer selection affects generalization is a research direction worth exploring (Appendix C.2).

    Scope of experiments

    • The model under test is only Llama3.1-70B-instruct, and all fine-tuning and probe experiments are conducted on it (§7).
    • Defense evaluation covers four types: probe, OpenAI moderation API, claude-3-sonnet-20241022 frontier monitoring, and self-reflection (§7).
    • Cipher coverage is 7 ciphers in 4 families: Walnut50/51/52, EndSpeak/StartSpeak, ASCII, Keyed Polybius; OOD testing includes 1 unseen cipher family (§5.2, §7).
    • The paper does not report the sample size of each evaluation set, the number of experiment repetitions, or confidence intervals.
    • The paper does not report specific numbers for the probe's inference overhead, stating only that the computational overhead is small (§1).

  6. Summarize the main content of the paper

    CIFR benchmark plus activation probe: the authors report that the probe detects cipher fine-tuning attacks with over 99% accuracy and generalizes to unseen cipher families, and that harm is distributed across multiple directions.

    A defense paper: for cipher steganography attacks on fine-tuning APIs (such as CMFT), the authors formalize the problem of "how a provider can defend against cipher attacks while preserving benign fine-tuning functionality," and provide one benchmark and one defense.

    • Problem addressed: attackers can encode harmful content into seemingly harmless fine-tuning data to bypass human inspection and content filtering; existing dataset-level monitoring fails against this.
    • Method highlights: the CIFR benchmark contains 6 benign fine-tunings (LIMA, OASST2, ProteinLMBench, Pure-Dove, MMLU, etc.) and 7 cipher attack fine-tunings (Walnut, EndSpeak, StartSpeak, ASCII, Keyed Polybius); some ciphers and entire cipher families are only in the test set; the defense is a linear probe jointly trained on the last-token activations at layer 32 of each fine-tuning instance of Llama3.1-70B-instruct.
    • Main results (Table 2): the probe achieves 1.000 accuracy on in-distribution tasks, 0.968 on the unseen same-family cipher Walnut52, and 0.995 on the unseen cipher family Keyed Polybius; OpenAI Moderation is 0.000–0.002 on all cipher tasks, and self-reflection and frontier moderation reach at most 88.5% on EndSpeak.
    • Mechanism finding: the authors state that cipher harm is distributed across multiple orthogonal directions, the first 100 directions all maintain perfect predictive accuracy, and steering experiments show that each direction can independently trigger jailbreaks, suggesting that single-direction safety mechanisms may fail.
    • Authors' conclusion: the authors believe the probe monitor tracks stable computational features preserved across fine-tuning, is low-cost, and is suitable for practical deployment; they also acknowledge that adaptive attacks were not tested and that benchmark coverage is incomplete, and recommend using CIFR together with other safety evaluations.

Related papers

Both use: ProteinLMBench
Both use: Alpaca
Show 3 more papers
Both use: Alpaca
Read originalarxiv.org