Language, Jailbreak, and Conciseness: A Training-Free Combination Study of LLM Attribute Steering Vectors

Can independently extracted language and behavior steering vectors be added layer-wise in a single forward pass while preserving each effect?

• Scenario: The authors state that activation steering adds contrast activation differences to the residual stream at inference time, which is lighter-weight than prompting or fine-tuning; single attributes are relatively mature, but real deployments often need to satisfy multiple attributes simultaneously.
• Existing gaps: The authors state that evidence for additive combinations is contradictory: some report destructive interference, others report mild success, and none attribute the cause to layer, strength, or geometry; subsequent work has largely shifted to combinations requiring training, abandoning the training-free advantage.
• Core observation: The authors believe unreliability may stem from unspecified layer and strength configurations; refusal is described as a single direction that is nearly universal across languages, so language vectors may combine cleanly with jailbreak vectors. Conciseness is treated as a second abstract behavior not previously tested.
• This paper's approach: On four instruction-tuned models across two model families and two scales, they characterize language, jailbreak, and conciseness separately, then perform two- and three-attribute additive combinations, linking behavioral results to approximate orthogonality in the residual stream.

Single-attribute steering

• Rimsky et al. (2024) et al.: additive intervention on intermediate residual streams; vectors are usually difference-in-means of contrast conditions (Marks and Tegmark, 2024). The paper lists its control of sentiment, honesty, toxicity, refusal, and language (Li et al., 2023; Turner et al., 2023; Arditi et al., 2024; Konen et al., 2024; Gurgurov et al., 2026). The authors state these results only show single attributes are reliable and do not indicate what happens when multiple vectors are injected simultaneously.

Training-free additive composition

• van der Weij et al. (2024): sums multiple abstract safety concept vectors and injects them at once; the authors state this is largely unsuccessful, with destructive interference between behaviors.
• Stolfo et al. (2025): injects at different layers separately, composing two instruction-following constraints; the authors state this is only a side experiment on a single model and small prompt set.
• Scalena et al. (2024) and Cao et al. (2024): the former extracts and composes language, safety, and formality by attention head across all layers, which the authors state is fundamentally different from single-layer intervention; the latter uses bidirectional preference optimization to obtain vectors and then performs partially successful additive composition of wealth and power at a single layer. The authors state none of these works trace success or failure to layer or head granularity, strength, or attribute geometry, and most do not go beyond a single model and single language.

Learnable composition

• Radevski et al. (2026), Han et al. (2024), Nguyen et al. (2025): learn composition on input tokens, linear mappings of final hidden states, and per-token gates on activation space, respectively; the latter includes explicit orthogonality and sparsity objectives. The authors state all three require gradient-based training for each attribute.

Baselines and data

• Baseline methods: the same base prompt plus an explicit instruction suffix; three suffixes per target for single attributes, and for composition the best-performing suffix for each target is used (Appendix A).
• Benchmarks/datasets: language extraction uses FLORES; jailbreak extraction and evaluation use Arditi et al. (2024); conciseness extraction uses instruction-removed prompts from IFEval (Zhou et al., 2023; Stolfo et al., 2025); language and conciseness evaluation use 70 English prompts from CLaS-Bench (Gurgurov et al., 2026).

The authors position this paper as: explaining when training-free addition succeeds or fails, composing multilingual attributes with abstract behaviors through layer-wise intervention, and linking results to vector geometry across model families and scales.

Compositional steering first extracts a unit-normalized DiffMean vector for each attribute separately, then adds them at their respective layers in the same forward pass.

Problem setup and extraction

• Three attributes: language L steers generation from English to non-English; jailbreak J makes the model comply with harmful instructions; conciseness C makes outputs shorter. Attributes are umbrella terms; behavior refers specifically to jailbreak and conciseness.
• Extraction location: for a chat-formatted instruction x, take the residual stream activation hl(x) at layer l at the last token after the instruction and before assistant generation, without pooling over the sequence.
• DiffMean: the mean activation for condition c is the average of hl(x) over the instruction set for that condition. The vector from condition b to a is the difference of the two means, then divided by its norm to obtain a unit vector, so that strength α alone determines the intervention magnitude. The contrast conditions are English vs. target language, harmful vs. harmless, and with vs. without a short suffix.

Application and composition

• Single vector: at layer l, change the hidden state to hl plus α times v(l), then pass it to subsequent layers.
• Multiple attributes: each vector is added only at its own layer. If vectors from multiple layers fall on the same layer, first multiply each by α and then sum before adding.
• Layers and strengths: single attributes are scanned over four depths and five α values. For Llama, α is 1.0, 2.0, 4.0, 6.0, 8.0; for Qwen, it is 10.0, 20.0, 40.0, 60.0, 80.0. The authors state that the raw magnitudes of the two families of vectors differ, so they adjust per family. For composition, they take the relative layers that are best or near-best for each model–attribute in the single-attribute setting, and α is selected separately by model and attribute (Table 5, Appendix D).

Data scale

• Extraction: FLORES ten languages (Arabic, German, English, Spanish, French, Korean, Chinese, Russian, Japanese, Portuguese) with 260 items each; jailbreak uses all 260 harmful samples against 260 random harmless samples; conciseness uses 540 pairs of prompts from IFEval with instructions removed.
• Evaluation: language and conciseness use 70 English prompts from CLaS-Bench; jailbreak uses 70 harmful prompts from the Arditi et al. (2024) test set. Each condition generates 70 responses. With conciseness, at most 512 new tokens; other conditions use 64.

Success determination

• Aggregate first, then harmonic mean: Table 2 specifies the metrics used for each setting; first aggregate across generations, then combine with the harmonic mean to penalize configurations that improve only one of the objectives.
• LFS: fastText gives the language distribution; success is recorded when the target language probability exceeds 95%, and LFS is the proportion of successful responses.
• OR: Qwen3-8B scores 0, 1, or 2 (irrelevant or uninterpretable, partially relevant, clearly relevant and coherent), then normalized to [0, 1]. Appendix B gives the full review prompt; the reviewer does not judge whether steering succeeded.
• JBS: Qwen3Guard-Gen-8B classifies responses as Safe, Controversial, or Unsafe, mapped to 0, 1, 2 and normalized to [0, 1], with higher meaning stronger jailbreak.
• CCS: 1 minus min(Lsteered/Lbaseline, 1). Length is counted in tokens using the corresponding model's tokenizer; if not shorter than baseline, it gets 0. For the three-attribute setting, the authors use Llama-3.1-8B outputs under the language–jailbreak two-attribute setting with at most 512 tokens as the shared length reference for all models.

Experimental Setup

• Models: Llama-3.1-8B-Instruct, Llama-3.1-70B-Instruct, Qwen2.5-14B-Instruct, Qwen2.5-32B-Instruct. The same model is used for activation extraction, steering, and generation.
• Layers: Llama-3.1-8B: 6, 14, 22, 30; 70B: 16, 36, 56, 76; Qwen2.5-14B: 10, 22, 34, 46; 32B: 12, 28, 44, 60 (Figure 2).
• Combination configurations: In Table 5, language layer/α are 8B: 6/4, 70B: 16/4, 14B: 10/40, 32B: 12/80; jailbreak: 14/6, 36/8, 22/80, 28/80; conciseness: 14/6, 36/8, 22/60, 28/80.
• Metrics: Language uses harmonic mean of LFS and OR; jailbreak uses JBS and OR; conciseness uses CCS and OR; combinations add the corresponding behavior metric before taking the harmonic mean.
• Evaluation: OR uses Qwen3-8B; JBS uses Qwen3Guard-Gen-8B; LFS uses fastText with threshold 95%.
• Baselines and sample size: Three explicit instruction suffixes; combinations concatenate each attribute's best suffix. 70 prompts per condition. Shading in Figure 2 is ± standard deviation; the paper does not report number of repetitions.

Main Results

The text and Figures 2–4 do not provide scores that can be transcribed cell by cell. Figure 2's y-axis is harmonic mean, range 0 to 1, x-axis is the layers above; solid lines are each α, dashed lines are the three prompt baselines.

Setting	Authors' description of Figure
L (Figure 2a)	Effective, but varies greatly with model, layer, and α; latest layer near 0 for all models and α
J (Figure 2b)	Generally peaks at the second earliest evaluated layer; Qwen2.5-32B slightly higher at layer 44 than layer 28
C (Figure 2c)	Also generally higher at the second earliest layer; Qwen2.5-32B slightly higher at layer 44 than layer 28
L+J (Figure 3a)	Average across four models higher than each prompt baseline; 70B has largest and most consistent gap; 8B has largest dispersion among language pairs
L+C (Figure 3b)	Average below prompt baselines, with larger variance across language pairs for four models; 8B weakest, 14B strongest and most language pairs higher
L+J+C (Figure 4)	Overall above prompt baselines for four models, with variation across language pairs greater than both two-attribute combinations

• Single-attribute interpretation: The authors believe success requires appropriate layer and strength, not simply increasing α; jailbreak and length control are more effective before late generation, while language favors earlier layers.
• Two-attribute interpretation: The authors state that combining language and jailbreak at selected layers and strengths yields joint performance stronger than explicit prompts. L+C shows more variation, which the authors attribute to weaker orthogonality between the two at later layers compared to L+J, with some overlap on certain languages and possibly greater sensitivity to tokenization.
• Three-attribute interpretation: The authors treat this as exploratory and state that increased variation is consistent with allocating three attributes across a fixed set of layers.

Vector Geometry

• Cross-language: In Figure 5a, pairwise cosine similarities of different language vectors are high at early layers (about 0.6–1.0) and decline toward later layers. The authors state that language attributes are more aligned early and more divergent near the output.
• Cross-attribute: In Figures 5b–d, cosine similarities between average language vector and jailbreak, conciseness, and between jailbreak and conciseness are near 0 at all depths, with no consistent positive or negative trend. The authors state that the three occupy approximately orthogonal subspaces. Partial exceptions are slight positive correlation between language and conciseness at later layers, higher for languages tokenized into more tokens; the authors suggest this may partly reflect sequence length rather than a shared behavioral direction.
• Magnitude: Figure 6 shows norms before normalization increase with relative depth. At early layers, all three attributes are small and similar; afterward, language and jailbreak grow more, while conciseness remains smaller. Absolute scales differ between Llama and Qwen. The authors state that after normalization, raw norms do not directly determine α.

Other Ablations and Analyses

• Per-language main results: Appendix F.1 reports results broken down by target language; the main text does not provide transcribable scores.
• Per-language cosine: Appendix H and Figure 20 give depth curves for each language vector against jailbreak and conciseness vectors.
• Combinations and similarity: Appendix I, Figure 21 compare single, two, and three-attribute scores by language pair. The authors state that combinations that clearly degrade relative to language-only steering often correspond to behavior vectors with cosine further from 0 with language vectors. Example: Qwen2.5-32B's L+J is below L-only only for German and Russian, which have the highest language–jailbreak cosine at later layers. The authors call this preliminary support; the relationship is not perfect.
• Table 1: Provides qualitative examples of success; three-attribute examples come from Qwen2.5-14B, others from Llama-3.1-8B. Appendix C provides English translations.
• Negative exceptions: Language steering at the latest layer is near 0; L+C averages below prompt baselines; Qwen2.5-32B's jailbreak and conciseness peaks are not at the second earliest layer.

Limitations and future directions noted by the authors

• Model type: Experiments are limited to instruction-tuned models. Extraction depends on the token position after the instruction, especially for jailbreak; the authors state the results may not directly generalize to base models or different prompt formats, and the most effective layers may differ for these models (Limitations).
• Fixed strength: Each experimental condition uses a constant intervention strength; the authors state the optimal strength may need to vary with context and be adjusted dynamically rather than fixed in advance (Limitations).
• Geometric relationship: Appendix I states the relationship between orthogonality and composability is only preliminary support, not perfect, and a more systematic characterization is left for future work.

Experimental coverage

• Models: Four instruction-tuned models, from the Llama-3.1 and Qwen2.5 families, at two scales (Section 4.1).
• Attributes and combinations: Single attributes L, J, C; two-attribute combinations L+J and L+C; and exploratory L+J+C (Section 5).
• Evaluation scale: 70 prompts per condition; language extraction covers ten languages, 260 each (Sections 4.1, 4.3).
• Geometry: Reports cross-language and cross-attribute cosine similarities, and the change in pre-normalization norm with relative depth (Section 6, Figure 5, Figure 6).
• Not reported in the paper: The number of repetitions per condition; the specific scores for Figures 2–4 are also not listed in the main text.

The authors study whether language, jailbreak, and conciseness steering vectors can be added without training in four instruction-tuned models.

• Problem: Single-attribute steering can control behavior alone, but evidence for injecting multiple vectors at once is inconsistent and often replaced by learnable combinations.
• Method: Contrast mean difference at the last instruction token and unit-normalize; single attributes scan four layers and five α values; combinations add each vector at its own optimal layer. Metrics are aggregated first, then harmonic mean of LFS, JBS, CCS, and OR.
• Single attributes: The authors state all three attributes are effective, but only at appropriate layer–strength combinations; jailbreak and conciseness favor the middle evaluated layers, language favors earlier layers, and language scores at the latest layer are near 0.
• Combinations: The authors state L+J averages above prompt baselines across four models, L+C averages below baselines with more dispersion across language pairs, and three-attribute overall still above prompt baselines but with more variation.
• Geometry and conclusion: The authors state these vectors are approximately orthogonal in the residual stream; the closer cosine is to 0, the more stable the combination. Slight positive correlation between language and conciseness at later layers is a partial exception. The authors argue layer-wise addition thus avoids destructive interference reported elsewhere. Limitations are coverage of only instruction-tuned models and fixed strength.