Depth Charge: Jailbreaking Large Language Models from Deep Safety Attention Heads

1. 01What problem does this paper attempt to solve?

   Shallow jailbreaks of open-source LLMs are easily blocked by alignment; SAHA instead performs white-box perturbation on deep safety attention heads.

   Open-source LLMs, even after alignment, can still be jailbroken by attacks from deep attention heads.

   • Scenario: The authors state that OSLLMs (e.g., Llama, Qwen) have publicly available weights and architecture, allowing malicious users to generate unsafe content; therefore, pre-release penetration testing is considered necessary by the authors to discover vulnerabilities and supplement alignment data.
   • Existing shortcomings: The authors categorize existing jailbreaks into prompt-level and embedding-level, claiming they target shallow layers and are easily blocked by shallow safety alignment. Examples given: random token dropping checks can defend against prompt-level attacks (Cao et al., 2024), and layer-specific semantic editing can counter embedding-level attacks (Zhao et al., 2024). The authors believe this creates a false sense of security.
   • Observation: The authors claim deeper attention layers are more vulnerable to jailbreaks, and neglected attention heads can be exploited. Their research question is: when attacking from deeper layers, are open-source LLMs still safe? The authors answer no.
   • Method: They propose SAHA. AIR uses ablation drop to locate safety-related heads, and LWP applies minimal perturbations per layer to elicit unsafe generations.
   • Threat model:
     • Goal: The output y is semantically aligned with the malicious target prompt p_t, and the external judge J(y)=0; the internal safety classifier must classify the perturbed embedding as benign, i.e., f_cls(e+p)=1.
     • Knowledge: The method assumes access to internal representations and the ability to train a safety classifier. Limitations state it is white-box overall, requiring internal attention mechanisms and gradients.
     • Capability: Add perturbation p to the malicious embedding e, e′=e+p. Total magnitude does not exceed ε, the component on heads strongly coupled to safety mechanisms does not exceed η, and η is much smaller than ε. The paper does not provide experimental values for ε and η.
     • Victim system: Open-source LLMs that have been instruction-tuned and aligned. Experiments use Llama3.1-8B-Instruct, Qwen1.5-7B-Chat, Deepseek-LLM-7B-Chat. Unsafe judgments in experiments use Llama-Guard-3-8B.

   Collapse this answer

2. 02What related research exists?

   Related work covers prompt-level and embedding-level jailbreaks and attention heads, and the authors position this paper at the head level.

   The paper organizes related work along two lines—attack surface and attention heads—and positions itself at the head level, deeper than embeddings.

   • Prompt-level jailbreaks: GCG (Zou et al., 2023) greedily searches replacements in the vocabulary; PAIR (Chao et al., 2023) uses an attacker LLM to rewrite rejected prompts; AutoDAN-Turbo (Liu et al., 2025) automatically searches for strategies and then uses them to generate adversarial prompts. The authors claim these methods only modify inputs, and their effectiveness degrades when simple alignment is applied at the encoder, citing Liu et al. (2024b) and Li et al. (2024): the former formalizes prompt injection and shows encoder-side defenses can reduce success rates, while the latter shows injected instructions in retrieved contexts can influence instruction-tuned models, and retrieval plus encoder-side mitigation can reduce exploitability.
   • Embedding-level and activation steering: SCAV (Xu et al., 2024) perturbs embeddings with concept activation vectors; CAA (Rimsky et al., 2024) uses the mean difference of contrastive activations as a steering vector. The authors cite Arditi et al. (2024) claiming a single refusal direction in the residual stream mediates safety: ablating this direction removes refusal, while amplifying it induces refusal. The authors cite ReFAT (Yu et al., 2024) claiming that reinforcing or purifying refusal features reduces the success rate of activation-level and input-level jailbreaks.
   • Attention heads: The authors cite Voita et al. (2019), Kovaleva et al. (2019), Marks et al. (2025), etc., claiming behavior is dominated by a sparse set of heads, with a few heads responsible for functions such as syntax and factual recall. PASTA (Zhang et al., 2023) takes the intersection of multi-task top-k heads and adjusts attention scores; Han et al. (2025) uses selective masking to filter cooperative heads. Sahara (Zhou et al., 2025b) uses representation shifts caused by ablation to find safety-critical heads, retaining heads within pathways and masking heads outside them to guide specific functions. The paper does not present Sahara as a jailbreak baseline.
   • Baselines and benchmarks: Experimental baselines are PAIR, GCG, AutoDAN (Liu et al., 2024a), AutoDAN-Turbo, SCAV, CAA, and ConVA (Jin et al., 2025). Benchmarks are JailbreakBench (Chao et al., 2024, 100 harmful behaviors) and MaliciousInstruct (Huang et al., 2024, 100 items, ten categories of malicious intent).

   The authors argue that after the attack surface moves from prompts deeper into embeddings, jailbreaks tend to be more effective because existing safety alignment reaches only limited depth. The authors claim this paper is the first to treat a single attention head as a structural, atomic causal unit, rather than only attacking deeper topological layers or the aggregated residual stream.

   Collapse this answer

3. 03How does the paper address this problem?

   SAHA uses AIR to select heads by ablation drop, then uses LWP to inject a closed-form minimal perturbation per layer.

   SAHA first locates safety-related attention heads, then adds perturbations to these heads' activations derived from the closed-form solution of a linear safety classifier, so that the internal probe judges the malicious embedding as benign, causing the model to generate outputs judged unsafe by the external judge.

   Problem setup

   • The jailbreak is formulated as constrained optimization of the pretrained model M_θ. The goal is for the output y to be semantically aligned with the malicious prompt p_t, with the external judge J(y)=0.
   • The malicious embedding e triggers refusal. The attacker constructs e′=e+p, hoping the internal classifier f_cls(e+p)=1, thus not triggering defenses.
   • Maximize semantic utility U under a total magnitude not exceeding ε; meanwhile J judges the output unsafe, and the perturbation component on safety-related heads does not exceed η, with η much smaller than ε.
   • The paper does not give the values of ε and η used in experiments. Figure 1 depicts the pipeline as: obtaining a steering vector from the linear separability of benign and malicious samples, AIR selecting heads, and LWP allocating per layer and injecting.

   AIR

   • Assumes access to a safety classifier f_cls trained on the target model's internal representations. The main text writes the data as D, with labels indicating whether the corresponding output is safe. Appendix C.1 implements a Latent Safety Probe with logistic regression: 200 items, 100 benign from alpaca-cleaned and 100 malicious from strongreject; takes the last token's hidden state and excludes prompts appearing in the evaluation benchmarks. Benign is labeled 1, malicious 0. The paper does not report the probe's accuracy.
   • For each head i, zero out its output and compare classification accuracy. Importance is Δ_i = Acc_orig − Acc⁽ⁱ⁾, the difference in accuracy before and after ablation. Take the top-k by descending difference.
   • To reduce sensitivity to a single sparsity, repeat selection over α in {0.25, 0.3, …, 1.0}, and use the selection frequency of head (ℓ, h) to form the global critical head set.
   • The authors claim AIR directly measures causal impact on safety. In contrast, APR only looks at single-head linear probe accuracy, not ablation drop.

   LWP

   • Each layer has its own budget. When layer ℓ has n_ℓ heads, the number of perturbed heads is k_ℓ = floor(α · n_ℓ), taking the top k_ℓ within the layer by AIR score, then merging across layers. α is in (0, 1].
   • The unit direction v on the selected set is added to the concatenated activations; the perturbed representation is the original activation plus εv. ε controls magnitude.
   • The linear classifier parameters are (w, b). The text writes attack success as cross-entropy below a threshold L_0, and defines P_0 as e to the power of −L_0. The minimal magnitude satisfies ϵ ≥ [log(P₀/(1−P₀)) − (w^⊤ e + b)] / (w^⊤ v), with the optimal v being the normalized projection of w onto the selected coordinates. This formula is a lower bound on the magnitude needed to push the linear probe across the decision boundary.
   • The paper does not report the P_0, L_0, or default α used in experiments. Appendix A derives the minimal perturbation from a first-order expansion of the target class cross-entropy, then approximates back to logit form in the linear region of the sigmoid. In contrast, GWP treats all heads as one pool, with k = floor(α · N).

   Injection and evaluation

   • Perturbations are added to the activations of the selected heads. The authors claim this concentrates attack capacity on safety-related computation and reduces collateral interference with semantic processing.
   • In experiments, evaluation is separate from the optimization objective: ASR checks whether Llama-Guard-3-8B labels the output unsafe; BERTScore uses Sentence-BERT to measure semantic similarity between prompt and response, taking the arithmetic mean of token similarities.
   • Section 5.3 states AIR+LWP is the default configuration, but does not specify the single α corresponding to the main table.

   Collapse this answer

4. 04What experiments did the paper conduct?

   On three open-source models and two benchmarks of 100 items each, SAHA's ASR in Table 1 is higher than all listed baselines.

   Experimental Setup

   • Models under test: Qwen1.5-7B-Chat, Llama3.1-8B-Instruct, Deepseek-LLM-7B-Chat (written as Deepseek-llm-7B-Chat in Table 1). All three are decoder-only Transformers that were instruction-tuned and aligned before release; the paper states they differ in scale, head configuration, and alignment strategy, but the main text does not specify the number of heads.
   • Benchmarks: JailbreakBench consists of 100 different harmful behaviors; MaliciousInstruct consists of 100 harmful instructions uniformly distributed across ten malicious intent categories, with examples mentioned in the text: sabotage, theft, hacking. The probe additionally uses alpaca-cleaned and strongreject, each with 100 items, and excludes prompts from the evaluation benchmarks (Appendix C.1).
   • Baselines: Prompt-level PAIR, GCG, AutoDAN, AutoDAN-Turbo; embedding-level SCAV, CAA, ConVA. Table 1 also has a Based row, but the paper does not explain its definition. Ablation comparisons are APR and GWP.
   • Metrics: ASR is the proportion of outputs judged unsafe by Llama-Guard-3-8B. BERTScore is computed by Sentence-BERT as the semantic similarity between the prompt and the generated response, taking the arithmetic mean of token-level similarities.
   • Evaluation: The unsafe judgment model is Llama-Guard-3-8B. The paper does not specify the judgment threshold, number of repetitions, or the α, ε, η, P_0 used in Table 1. The attacker LLM for PAIR is not specified.
   • Probes: A logistic regression LSP is trained separately for each victim model, with labels benign=1, malicious=0. The paper does not report probe accuracy.

   Main Results

   According to Table 1. Cells are ASR / BERTScore. The last column is the non-SAHA row with the highest ASR in that table, not the original column name in the paper.

   Swipe horizontally to read the table

   Model	Benchmark	SAHA	Other row with highest ASR
   Llama3.1-8B-Instruct	JailbreakBench	0.85 / 0.76	PAIR 0.57 / 0.07
   Llama3.1-8B-Instruct	MaliciousInstruct	0.87 / 0.84	SCAV 0.68 / 0.82
   Qwen1.5-7B-Chat	JailbreakBench	0.82 / 0.75	SCAV 0.71 / 0.73
   Qwen1.5-7B-Chat	MaliciousInstruct	0.86 / 0.81	ConVA 0.85 / 0.4
   Deepseek-llm-7B-Chat	JailbreakBench	0.91 / 0.70	SCAV 0.85 / 0.71
   Deepseek-llm-7B-Chat	MaliciousInstruct	0.81 / 0.77	SCAV 0.74 / 0.80

   • The authors claim SAHA consistently outperforms baselines in both ASR and semantic fidelity, and is stable across architectures. In the six ASR cells of Table 1, SAHA is higher than the listed baselines; there are counterexamples in BERTScore: on Llama3.1's JailbreakBench, SCAV is 0.77 vs SAHA 0.76; on Deepseek's JailbreakBench, SCAV is 0.71 vs SAHA 0.70; on Deepseek's MaliciousInstruct, SCAV is 0.80 vs SAHA 0.77.
   • The authors claim prompt-level methods are unstable across models, and embedding-level methods often trade off between success rate and semantics. In Table 1, ConVA on Deepseek's MaliciousInstruct is 0 / 0; AutoDAN-Turbo on Deepseek's JailbreakBench is 0.13 / 0.11.
   • The abstract claims SAHA improves ASR by 14% over SOTA baselines, without specifying model, benchmark, or comparison. Table 2's AIR+LWP and Table 1's SAHA are inconsistent on some BERTScore values: Qwen's MaliciousInstruct is 0.82 vs 0.81, Deepseek's JailbreakBench is 0.72 vs 0.70, Deepseek's MaliciousInstruct is 0.79 vs 0.77. The paper does not explain this discrepancy.

   Head Selection and Allocation Ablation

   • Table 2 compares APR/AIR with GWP/LWP. The authors claim that even with LWP, APR's ASR is lower than AIR, so probe-based head selection is inferior to ablation-based selection.
   • Counterexample: On Deepseek-llm-7B-Chat's MaliciousInstruct, APR+LWP has ASR 0.82, AIR+LWP has 0.81 (BERTScore 0.78 and 0.79 respectively).
   • On Llama3.1's MaliciousInstruct, AIR+GWP is 0.62 / 0.75, AIR+LWP is 0.87 / 0.84. The authors claim switching to global allocation harms semantic fidelity and reduces ASR in some settings, citing Qwen1.5 as an example.

   Retention Ratio and Magnitude

   • Figures 3–4: The authors claim ASR increases monotonically with α for all models and strategies. Llama3.1 has a threshold around α ≈ 0.45; except for AIR+GWP, Qwen and Deepseek gradually saturate from a higher starting point. The authors claim that when α is between 0.5 and 0.7, Qwen and Deepseek are near their peak, while Llama needs to cross its own threshold. The converted text does not show specific ASR values on the curves.
   • Figures 2 and 5: The authors claim that as α decreases, the required perturbation magnitude per head increases. In the extracted text, the x-axis of these two figures is labeled 0.3 to 1.0, the y-axis is magnitude, and the legend shows three models; the paper does not give endpoint values in the main text.

   Location of Safety-Critical Heads

   • Figure 12 shows the selection frequency of each head under AIR+GWP; the caption states that color indicates selection frequency across all α, with red circles marking the 10 most frequently selected heads. The converted text does not show colors or red circle positions.
   • The authors claim the last head, Head 31, is prominent across models; Deepseek's critical heads are more dispersed in the middle-to-upper layers, Llama3.1's around layer 7, Qwen1.5's around layer 5 with additional lower-layer activations. The authors believe the internal routing of safety signals varies by model, so they prefer layer- and model-specific hardening rather than a uniform shallow defense.

   Other Ablations and Analyses

   • The introduction claims effectiveness under low perturbation budgets and against composite defenses; the experimental section does not provide budget values, defense names, or corresponding ASR.
   • Section 5.3 states that in Figures 6–11, LWP is more uniform, while GWP is disproportionately concentrated in shallow layers; Section 5.4 states that AIR+GWP concentrates large perturbations in deeper layers as α shrinks. The two descriptions differ, and the paper does not explain.
   • Appendix Figures 13–14 compare the Jaccard similarity of head sets as α varies; the main text does not give readable specific similarity values.
   • Appendix C.1's probe is a logistic regression on 200 items per model; the paper does not report probe accuracy.
   • The Based row in Table 1 is undefined in the paper. On JailbreakBench, its ASR is 0.06, 0.06, 0.29 (Llama3.1, Qwen, Deepseek).
   • The introduction mentions statistical testing, but the main text does not provide test statistics.

   Collapse this answer

5. 05What are some points that can be further explored?

   The authors note that SAHA requires white-box gradients, and the head selection strategy targets Transformers.

   Limitations and Future Directions Noted by the Authors

   • White-box (Limitations): Requires internal attention mechanisms and gradients, so it cannot be directly applied to black-box commercial APIs. The authors state this setting is intended for developers and safety researchers to conduct pre-release red-teaming and mechanism analysis of open-source weights.
   • Architecture (Limitations): The method targets Transformer attention heads. The authors state that if moving to other architectures (e.g., state-space models), the form of safety-critical components may change, and head selection strategies may need adaptation.
   • Alignment (Conclusion): The authors state that alignment techniques capable of defending against vulnerabilities in deeper representations are needed.

   Experimental Coverage

   • Models under test are Llama3.1-8B-Instruct, Qwen1.5-7B-Chat, Deepseek-LLM-7B-Chat, totaling 3 (Section 5.1).
   • Attack evaluations are JailbreakBench and MaliciousInstruct, both described as 100 items; the latter covers ten malicious intent categories.
   • Comparisons include PAIR, GCG, AutoDAN, AutoDAN-Turbo, SCAV, CAA, ConVA, as well as APR/AIR, GWP/LWP, and retention ratio α (Table 1, Table 2, Figures 3–4).
   • Unsafe judgment uses Llama-Guard-3-8B, semantic metric is Sentence-BERT's BERTScore. Probe data are alpaca-cleaned and strongreject, each 100 items (Appendix C.1).
   • The paper does not report the number of repetitions, Llama-Guard threshold, α, ε, η, P_0 for Table 1, PAIR's attacker LLM, settings and results for composite defenses, or the statistical testing values mentioned in the introduction.

   Collapse this answer

6. 06Summarize the main content of the paper

   SAHA performs white-box jailbreaks from attention heads; in Table 1, the ASR for all three models is higher than the listed baselines.

   SAHA is a white-box attention-head jailbreak for aligned open-source LLMs.

   • The authors' question is: after prompt-level and embedding-level attacks are blocked by shallow alignment, can attacking deeper attention heads still elicit unsafe outputs?
   • AIR selects heads using the ablation drop of a linear safety probe and counts selection frequency across multiple retention ratios; LWP allocates the budget per layer to the top-ranked heads within each layer, then injects a closed-form minimal perturbation along the projection of the classifier weights.
   • In Table 1, the ASR is the highest among the listed methods across two benchmarks and three models: on JailbreakBench, Llama3.1-8B-Instruct is 0.85, Qwen1.5-7B-Chat is 0.82, and Deepseek-llm-7B-Chat is 0.91; on MaliciousInstruct, they are 0.87, 0.86, and 0.81, respectively. BERTScore has cells lower than SCAV.
   • The authors claim the head level is riskier than shallower attack surfaces, and that checking only inputs or shallow representations is insufficient; these heads should be aligned or monitored. Limitations states this depends on white-box access and targets Transformers.

   Collapse this answer

Related papers