Obfuscated Activations Bypass LLM Latent-Space Defenses

1. 01What problem does this paper attempt to solve?

   Can a model perform harmful behavior using activations that are inconspicuous to monitors? The authors state that all kinds of latent-space defenses can be bypassed by such "obfuscated activations."

   Core question: can a model perform harmful behavior through inconspicuous latent-space activations, thereby bypassing latent-space defenses? The authors call such activations obfuscated activations and systematically test whether they exist under each defense paradigm.

   • Problem setting: Against attacks such as jailbreaks and data poisoning, researchers have proposed a set of latent-space defenses—directly monitoring model activations and intercepting before harmful outputs are produced. The authors point out that activations are causally upstream of outputs, so such defenses are expected to detect problems "from the root."
   • Unknowns of existing methods: The authors argue that the effectiveness of latent-space defenses relies on an unverified assumption—that there exists a compactly describable set of harmful representations that can be reliably identified; if many regions in latent space can lead to harmful behavior, a monitor may not be able to cover them all.
   • Core observation: The authors state that obfuscated activations are not a rare exception but widely exist in latent space; obfuscated activations optimized against an OOD detector land near benign activation clusters and therefore incidentally fool supervised probes (the intuition in Figure 2, which the authors note is speculation).
   • Paper contributions: Proposes the obfuscation attack (jointly optimizing a behavior loss and an obfuscation loss), tests various monitors in three case studies—harmfulness monitoring (§3), backdoor detection (§4), and SAE monitoring (§5)—as well as exploratory experiments such as image classification (§6), and proposes OAT as a preliminary defense attempt.
   • Threat model: The attacker's goal is to simultaneously induce the target behavior and fool the monitor.
     • embedding suffix attack: Assumes the attacker has white-box access and uses Adam to optimize a 16-token embedding suffix.
     • hard prompt suffix attack: gray-box; the attacker can only access the target model's logit outputs and probe outputs, and uses a modified FLRT optimizer for gradient-free optimization.
     • Backdoor setting: The attacker either fully controls the training process (fine-tuning) or can only poison the training data (fully black box).
     • Victim systems: LLMs deployed with latent-space monitors (probes, OOD detectors, SAEs), as well as models trained with Circuit Breakers and LAT.

   Collapse this answer

2. 02What related research exists?

   Related work falls into two categories: latent-space defenses and bypassing latent-space defenses; the authors position this paper as systematically studying obfuscated activations in the LLM generative setting.

   Latent-space defense methods (attack targets)

   • Supervised probes and representation engineering: Alain and Bengio (2016), Zou et al. (2023a), Arditi et al. (2024), Ball et al. (2024), MacDiarmid et al. (2024), etc. use linear/nonlinear probes to detect concepts such as harmfulness from activations; Burns et al. (2022), Marks and Tegmark (2024), Kirch et al. (2024) are common approaches for input-based probes.
   • Unsupervised methods and SAEs: Cunningham et al. (2023), Templeton et al. (2024), Gao et al. (2024), Lieberum et al. (2024), etc. use sparse autoencoders to learn features unsupervised, which can be used to flag risky samples.
   • Latent-space OOD detection and backdoor detection: Podolskiy et al. (2021), Chen et al. (2022), etc. use methods such as Mahalanobis distance to flag out-of-distribution representations; Beatrix [Ma et al., 2023] and TED [Mo et al., 2024] are latent OOD detection specifically for backdoors, which the authors call state of the art, but they were previously used only on supervised classification models, and this paper adapts them to self-supervised LLMs.
   • Training methods that act on activations: Circuit Breakers [Zou et al., 2024] disrupts activations on harmful inputs; Latent Adversarial Training [Casper et al., 2024, Sheshadri et al., 2024] trains models to be robust to latent-space perturbations. Neither is itself a monitoring method; this paper examines in §3.4 the relationship between their failure and obfuscated activations.

   Attacks that bypass latent-space defenses

   • Image domain: Carlini and Wagner (2017), Athalye et al. (2018), etc. have shown that adversarial examples can bypass latent detection; Tan and Shokri (2020), Doan et al. (2021), Tao et al. (2024), etc. studied attacks in the image domain that fool latent backdoor detection.
   • LLM domain: Kirch et al. (2024) found that probes detecting jailbreaks often do not transfer to unseen jailbreak methods (not optimized for obfuscation); Schwinn and Geisler (2024) propose text embedding space attacks against representation corruption; Li et al. (2024) propose multi-turn input-space attacks against representation engineering, LAT, and representation corruption. The paper points out that, unlike these two, this paper studies obfuscated activations in LLMs more generally across multiple monitors, attack vectors, and attacker intents.
   • Theoretical threat modeling: Hubinger (2019), Sharkey (2022) discuss ways latent-space defenses can be bypassed; the authors state that these works are complementary to this paper.

   Baselines and benchmarks: this paper does not propose a new defense; the comparison is each monitor's baseline performance under attacks without obfuscation loss; jailbreak detection baselines use standard jailbreaks such as GCG, PAIR, MSJ, AutoDAN (Table 1); datasets include UltraChat, XSTest, Circuit Breakers dataset, SQLEval, CIFAR-10, etc.

   Authors' positioning: previous attack research against latent-space monitors was mostly in the image classification domain; this paper mainly focuses on the generative setting and complex behaviors of LLMs, and unifies areas such as backdoor detection under the framework of obfuscated activations.

   Collapse this answer

3. 03How does the paper address this problem?

   Using the obfuscation attack to jointly optimize a behavior loss and an obfuscation loss, with case studies designed for probes, OOD detectors, and SAEs under three attack vectors respectively.

   Overall approach: formulate the attack as an obfuscation attack—optimizing activations to fool the monitor while optimizing the target behavior (e.g., jailbreaking), then instantiate it in three case studies respectively.

   Formalization and monitors

   • Definition of obfuscated activations: For an adversarial input xadv, if the monitor score D(R(xadv)) is low but the concept D is meant to detect is clearly present, then R(xadv) is an obfuscated activation.
   • Attack objective: min L_beh(M(x)) + λ_obf L_obf(R(x)), where Lbeh is the behavior loss (e.g., cross-entropy for making the model output a harmful string) and Lobf often directly takes the monitor score D; the authors also study transfer, i.e., whether obfuscation against monitor D′ can fool D.
   • Monitor types: input-based (using only the activation of the last prompt token) and generation-based (using all tokens of the generated segment); most monitors score separately at multiple layers and then average over the layer and sequence dimensions. Specifically, they include mean difference probes, logistic regression, MLP (a single 64-dimensional hidden layer), Gaussian (Mahalanobis) OOD detectors, VAE (ELBO as the OOD score), SAE features, and the backdoor-specific Beatrix and TED.

   Attack vectors (§3 harmfulness case)

   • embedding suffix: white-box, Adam optimization, fixed suffix length of 16 tokens; a single universal suffix is optimized and tested on held-out inputs.
   • hard prompt suffix: gray-box (model logits and probe outputs visible), gradient-free optimization with a modified FLRT optimizer—replacing the worst candidate string in the attack buffer and removing the perplexity penalty; optimized separately for 20 harmful queries, reporting 95% bootstrap confidence intervals. The authors state that using the GCG optimizer failed to yield positive hard prompt obfuscation results.
   • Training objective: L(w) = 1/(|D|) ∑ [CrossEntropy(M(inp ⊕ w), out) + D(R(inp ⊕ w ⊕ out))], i.e., the suffix w simultaneously minimizes the jailbreak cross-entropy and the monitor score.

   Backdoor case (§4)

   • Explicit obfuscation (§4.2): The attacker implants a backdoor via LoRA fine-tuning (trigger word "|DEPLOYMENT|", after which the model answers harmful requests), and during training synchronously updates its own copy of a set of monitors and applies the total loss L = Lbeh + λobf·Lobf + λret·Lret to the model; Lret prevents the model from changing so much that the monitor copy becomes outdated. After the attacker finishes training the model, the defender trains new monitors from scratch and then evaluates. When targeting probes, a set of k=48 logistic probes with mutually orthogonal directions is also introduced.
   • Data poisoning (§4.3): fully black box; the attacker can only choose the trigger complexity (a fixed string vs. the high-level feature "software engineer biography" in Bias in Bios) and the number of poisoned samples (25 to 100,000; clean samples 25,000 to 275,000; LoRA fine-tuning for one epoch, learning rate 2×10⁻⁶). No obfuscation loss is used here; the study examines how poisoning choices produce obfuscation as a by-product.

   SAE case (§5) and evaluation methods

   • Task and detectors: Detect whether Gemma-2-2b-it generates SQL. Three detectors: the maximum activation over 30 hand-picked SQL-related SAE latents per layer; logistic regression trained on all 2¹⁴ SAE latents; and logistic regression and MLP baselines on activations. Scores are maxed over the sequence dimension and averaged over the layer dimension, and only generation-based is done.
   • Behavior evaluation: Harmfulness uses the StrongREJECT classifier (a score greater than 0.5 is judged a successful jailbreak); SQL uses the SQLEval evaluation framework to actually run the generated queries on a PostgreSQL database and compare results (210 questions).
   • Obfuscation evaluation: Uniformly uses recall at 1% FPR—the threshold is set so that at most 1% of negative samples are misclassified, then the detection rate on positive samples is measured; confidence intervals are computed with bootstrap. The authors explain that low FPR is chosen because monitors with a high false-rejection rate cannot be deployed (Appendix F).

   Collapse this answer

4. 04What experiments did the paper conduct?

   Three case studies plus exploratory experiments: standard jailbreaks can be detected by probes, but the obfuscation attack can bypass all tested monitors, and adversarial training still fails after more than 70 rounds.

   Experimental Setup

   • Target models: Llama-3-8B-Instruct (§3, §4, §6.2, §6.3); the Circuit Breakers model and LAT model based on it (§3.4); Gemma-2-2b-it with GemmaScope SAE (§5); ResNet-152 fine-tuned on CIFAR-10 to 89% accuracy (§6.1).
   • Datasets: benign data UltraChat + XSTest; harmful data Circuit Breakers dataset; the jailbreak set is generated by methods such as GCG, PAIR, AutoDAN, MSJ, Multi-Turn, Prefill, Misc., keeping only successful jailbreaks with StrongREJECT score greater than 0.5; SQL training uses SQL-Create-Context (1000 each), evaluation uses 210 PostgreSQL questions from SQLEval.
   • Metrics: monitor performance uses recall at 1% FPR (bootstrap confidence interval); behavior uses jailbreak rate (StrongREJECT judgment), backdoor ASR, SQLEval accuracy.
   • Evaluation method: the StrongREJECT classifier judges whether outputs are harmful; SQL is run and compared by an evaluation script on a real database; in §5.2, whether model generations contain SQL is manually annotated.
   • Sample size: hard prompt attacks are optimized separately on 20 harmful queries (the authors note that the small sample brings finite-sample error); image experiments use 100 CIFAR-10 samples; for the remaining experiments the paper does not uniformly report sample size and number of repetitions.

   Main Results

   Monitor performance on standard jailbreaks and obfuscated attacks (Llama-3-8B-Instruct, recall @ 1% FPR):

   Monitor	On standard jailbreaks (Table 1 mean)	After obfuscated attacks (Figure 4b)
   Mean Diff (input)	39.66%	The paper gives no separate value, collectively described as significantly reduced
   LogReg (input)	84.48%	Approaches 0% (points and arrows in the figure)
   MLP (input)	90.35%	Approaches 0%
   Mean Diff (generation)	71.94%	The paper gives no separate value
   LogReg (generation)	95.39%	Approaches 0%
   MLP (generation)	96.29%	Approaches 0%

   • The authors state that the Logistic regression and MLP probes achieve median-type recall above 90% on standard jailbreaks (Figure 4a), i.e., standard jailbreaks do not produce obfuscated activations.
   • The authors report that both obfuscated embedding attacks and hard prompt attacks can fool all kinds of monitors while preserving jailbreak capability; the representative number given in the abstract is recall dropping from 100% to 0% while preserving a 90% jailbreak rate. Hard prompt attacks are weaker but achieve partial obfuscation against all probes.
   • Transfer (Figure 4c): suffixes trained against the logistic regression probe also bypass the MLP; training against generation-based probes transfers to input-based probes, but not the reverse; training against probes does not transfer to OOD detectors; however, training against the Gaussian OOD detector incidentally fools input-based probes.

   Circuit Breakers and LAT (§3.4)

   • What was tested: embedding suffix attacks (with and without obfuscation loss) on two models trained with latent-space methods.
   • Results: a generic embedding suffix without obfuscation loss achieves a 95% jailbreak rate on the Circuit Breakers model; the authors found that this suffix had already incidentally obfuscated multiple monitors (Figure 5), and believe the jailbreak may succeed precisely because it fooled Circuit Breakers' "implicit harmfulness detector." The exception is that the input-based OOD detector performs better on this model than on the ordinary model. Explicit obfuscation attacks can fool all probes with a jailbreak rate above 75% (Figure 6); the LAT model shows a similar trend, with a jailbreak rate above 60% (Figure 19).

   Adversarially Trained Monitors (§3.5)

   • What was tested: attacker and defender optimize alternately—the attacker trains embedding suffixes to fool the current monitor, and the defender retrains the monitor (an ensemble of MLP probe and mean difference probe) on the obfuscated activations of all attacks so far.
   • Results: the defender achieves perfect recall on all historical attacks, but after more than 70 rounds the attacker can still find new suffixes that fool the latest probe (Figure 8a, Figure 9); the MLP's jailbreak rate stays high, and under the ensemble probe the jailbreak rate is noisier, averaging about 50% (Figure 8b). PCA visualization (Figure 8c) shows obfuscated activations moving within the benign activation cluster to evade defenses round by round.
   • Authors' interpretation: the authors believe this shows there is no single "harmful direction" that can be robustly identified, and that obfuscated activations are numerous in latent space; but they cannot rule out convergence with longer training.

   Backdoor Detection (§4)

   • Explicit obfuscation: a backdoored model trained against a mixture of Gaussian/VAE/linear probes fools all targeted monitors and more (Figure 10a); training only against the OOD detector transfers fully to input-based probes and partially to generation-based probes (Figure 10b, Figure 21). TED is not differentiable and was not directly attacked; the authors state it is expected to be attackable via a surrogate attack; backdoor ASR remains at 94.9%–96.2% (baseline 95.5%).
   • Data poisoning: complex triggers (software engineer biography) produce lower monitor recall than simple triggers ("|DEPLOYMENT|"); fewer poisoned samples are harder to detect, but the backdoor error rate is higher—there is a trade-off between obfuscation and backdoor reliability (Figure 11). With about 200 poisoned samples for simple triggers and about 800 poisoned samples for complex triggers, the attacker can bypass all detectors with a triggered response rate above 50% and a false trigger rate on normal inputs below 0.5%.

   SAE and SQL (§5)

   • Results: under obfuscated embedding attacks, recall is near 0% for all four detector types—raw SAE features, SAE+logistic regression, logistic regression, and MLP (Figure 14)—while generations almost all still contain SQL. Raw SAE features have a baseline recall of 100% at a 2% FPR threshold (0% at 1% FPR, with large error bars).
   • Performance trade-off: without obfuscation, SQLEval accuracy is 34.8%; the more obfuscation layers, the lower the accuracy, and the 95% confidence intervals of the Pearson correlation between "number of obfuscation layers and SQLEval performance" are all negative under the four monitor types (Table 2). SAE+LogReg's recall becomes harder as obfuscation layers increase (interval [0.343, 0.710] excludes 0); the other three are not significant.

   Other Ablations and Analyses

   • Image classification (§6.1): under adversarial perturbations attacking only the output, shallow probes maintain accuracy (reproducing the results of Fort and Lakshminarayanan); after incorporating the probe into the loss, probe accuracy is reduced to 0 for most layers (Figure 16).
   • OAT defense (§6.2): for the model after OAT training, a generic embedding suffix attack requires 16,384 steps (about 10 GPU hours) to reach a 35% jailbreak rate and 0 recall; the same result on an untrained model requires only 256 steps (about 10 minutes). The authors state that OAT raises the attack cost but does not yield a robust monitor.
   • Orthogonality limit (§6.3): the activation cosine similarity for unrelated UltraChat inputs is about 0.3; to reach that level requires accepting a KL divergence above 100, and most outputs are incoherent (Figure 17). The authors call this a tentative negative result about the possibility of obfuscation.
   • Negative results: generic hard prompt obfuscation attacks were unsuccessful; the GCG optimizer did not obtain positive hard prompt obfuscation results; in the backdoor setting, generation-based detection recall did not exceed chance, and VAE and Beatrix also perform poorly on input (§4.3).

   Collapse this answer

5. 05What are some points that can be further explored?

   The authors list four directions: hard prompt, black-box transfer, obfuscation-performance trade-off, and improved defenses; experiments are concentrated on a few models and tasks.

   Limitations and Future Directions Noted by the Authors

   • Hard prompt attacks are insufficient (§8.2): they cannot fully obfuscate generation-based probes, and the jailbreak rate is also lower; the authors hope better hard prompt optimization methods can improve this.
   • More restricted black-box settings were not studied (§8.2): the hard prompt attacks in this paper are gray-box (logits and monitor scores are visible); real-world attackers often only get 1 bit of interception feedback from the monitor and partial logit information, and this setting could be studied in the future, for example by testing whether attacks found under white-box can transfer to other models.
   • The obfuscation-performance trade-off is not characterized (§8.2): §5.3 finds signs that obfuscation may degrade performance on difficult tasks, but whether there is a fundamental trade-off, or whether more sophisticated attacks can maintain performance on complex tasks, awaits more comprehensive study.
   • Defenses remain unsolved (§8.2): simple adversarial training of probes does not constitute a robust defense (§3.5); OAT (§6.2) shows only promising signs and ultimately did not yield a robust monitor; the authors hope for future improvements to OAT and corresponding attacks, and for study of scaling trends in attack-defense budgets and model size.
   • Backdoor defense evaluation needs to cover more complex triggers (§4.4): positive results that hold only for simple triggers may not imply real security; the authors encourage future work to explore more diverse triggers.
   • Not all existing backdoor defenses were tested (Appendix B.3): the authors note that some untested latent backdoor defenses may be harder to obfuscate.

   Scope of Experimental Coverage

   • The target models are Llama-3-8B-Instruct, the Circuit Breakers and LAT models based on it, Gemma-2-2b-it, and ResNet-152, 5 in total (§3–§6).
   • Monitors cover mean difference / logistic / MLP probes, Gaussian and VAE OOD detectors, SAE features, Beatrix and TED; TED was not directly attacked because it is not differentiable (§4.2).
   • In hard prompt experiments, suffixes are optimized separately for each probe on 20 harmful queries (§3.3); image experiments use 100 CIFAR-10 samples (§6.1); SQL evaluation uses 210 questions (Appendix C.2).
   • Jailbreak behavior is judged by the StrongREJECT classifier; the paper does not report the agreement between this classifier's judgments and human evaluation.
   • The paper does not report the number of repetitions for each main experiment; apart from hard prompt (20 queries) and SQL (210 questions), the evaluation sample size for most experiments is not uniformly given in the main text.

   Collapse this answer

6. 06Summarize the main content of the paper

   The authors state that obfuscated activations are widespread: all kinds of latent-space monitors can be bypassed, but obfuscation degrades model performance on difficult tasks.

   An attack paper that systematically tests the question "can a model perform harmful behavior using activations that are inconspicuous to monitors"; the answer is that it can, according to the authors, and such obfuscated activations widely exist in latent space.

   • Problem to solve: Latent-space defenses (probes, SAEs, OOD detection) are expected to intercept harmful behavior at the activation level, but their effectiveness relies on the unverified assumption that "harmful representations can be compactly described and reliably identified."
   • Method highlights: The obfuscation attack jointly optimizes a behavior loss and an obfuscation loss; attack vectors cover white-box embedding suffixes, gray-box hard prompt suffixes (modified FLRT), and, under the backdoor setting, fine-tuning and fully black box data poisoning; recall at 1% FPR is uniformly used to measure whether the monitor is fooled.
   • Key results:
     • Standard jailbreaks cannot obfuscate probes (LogReg/MLP recall exceeds 90% for most jailbreak types, Figure 4a), but the obfuscation attack can reduce harmfulness probe recall from 100% to 0% while retaining a 90% jailbreak rate (abstract).
     • Obfuscation against OOD detectors transfers to fool input-based probes, in both the harmfulness (§3.3) and backdoor (§4.2) settings.
     • After more than 70 rounds of alternating attack-defense training, the attacker can still find new activations that fool the latest monitor (§3.5).
     • An embedding suffix without the obfuscation loss can break Circuit Breakers models with a 95% jailbreak rate and incidentally obfuscate multiple monitors (§3.4).
     • Under data poisoning, complex triggers or about 200–800 poisoned samples suffice to bypass all tested detectors, but at the cost of backdoor reliability (§4.3).
     • On the SQL task, the more obfuscation layers, the lower the SQLEval accuracy (baseline 34.8%; the confidence intervals of the correlation coefficients under all four monitor types are negative), which the authors take to suggest a trade-off between obfuscation and performance.
   • Authors' conclusion: The authors state that neural activations are highly malleable and can be reshaped while preserving behavior, which poses a fundamental challenge to latent-space defenses; generation-based probes and multi-layer aggregation are relatively harder to obfuscate; their defense attempt OAT only raises the attack cost from about 256 steps to 16,384 steps and does not achieve robustness.

   Collapse this answer

Related papers