Skip to content

Constructing a steering vector via input optimization

Creating Steering Vectors via Input Optimisation

arXiv: Jailbreaks, Prompt Injection, Poisoning & Defenses· arXiv:2611.17481· Batuhan AkarSecure and Private AI Lab; ETH ZürichIndexed by this site · 4 days ago· Originally publishedSelected Attacks· Its place in the scan record →SelectedAttention38

Quick read

Attacks

Based on the paper PDF (AI-generated); refer to the original

The authors propose ActGCG, using GCG to search for discrete suffixes that reproduce the activations of a steering vector; German behavior is reproduced relatively well, while the anti-refusal direction is weaker under HarmBench.

Threat modelThe authors study whether an input suffix can reproduce the behavior of a steering vector: the optimization stage requires white-box access to build the steering vector and read activations, while the generation stage requires no access to the model's internals. The anti-refusal direction targets open-source small models, with jailbreaks judged by the HarmBench classifier.

Questions
Activation steering requires modifying the model's internal activations during generation, which is infeasible when the model can only be accessed through a chat interface. The authors ask: can a single input suffix reproduce the behavior of a steering vector, thereby avoiding the need to access the model's internals during generation?
Methods
The authors propose ActGCG: replacing GCG's cross-entropy loss with the squared error against the "target activation after adding the steering vector," matched at filler token positions across four layers; they additionally extend it to a universal suffix, a model translation that mid-optimization uses a larger model to rewrite the suffix into readable text, and a soft suffix variant with continuous embeddings.
Experiments and results
German direction: discrete suffixes almost reproduce steering vector behavior (Gemma-2-2B 86.7%, Qwen3-1.7B 96.7%, 30 prompts), and the universal suffix can transfer to small models of the same family. The anti-refusal direction is weaker: phrase-based metrics are high (90.9%) but the HarmBench jailbreak rate is only 27.3% (Gemma-2-2B, n=22), and cross-model transfer is poor.
Limitations and future work
The authors note: only two concepts were tested, German and refusal; the scale is small (22–60 prompts), mostly a single seed; the discrete token space limits matching precision; the method is limited by the strength of the steering vector itself; the judge is imperfect, and phrase-based metrics are not a good proxy for safety.
Experimental setupGemma-2-2B-it、Qwen3-1.7B et al. · Dolly instruction dataset、Arditi et al. harmful/harmless instruction datasets · German success rate (langdetect)、HarmBench jailbreak rate et al.
Evaluated models
Gemma-2-2B-itQwen3-1.7BLlama-3.2-3BQwen3-4BGemma-1.1-2BGemma-2-9BQwen2.5-1.5BLlama-3.2-1B
Benchmarks
Dolly instruction datasetArditi et al. harmful/harmless instruction datasets
Metrics
German success rate (langdetect)HarmBench jailbreak ratephrase-based anti-refusal rateL2 distance / cosine to target activations
AI summaryand selection rationaleBatuhan Akar's bachelor's thesis proposes ActGCG, which changes GCG's optimization objective from a fixed output string to matching the activations after a steering vector is added to the residual stream, so that the steering behavior can be reproduced by an input suffix alone, without requiring access to the model's internals at generation time. On four small open-weight models including Gemma-2-2B and Qwen3-1.7B, the German suffix almost completely reproduces the steering behavior and transfers well across prompts and across models; the anti-refusal (jailbreak) direction has a high success rate under the metric of avoiding refusal phrases, but only partially succeeds under the stricter HarmBench classifier, and cross-model transfer is also poor. The thesis also proposes a universal suffix trained across prompts, a model-translation method that midway uses a larger model to rewrite the suffix into readable text, and a soft-suffix variant as an upper bound for matching. The author states that the purpose of this equivalence study is defensive.

Batuhan Akar's bachelor's thesis proposes ActGCG, which changes GCG's optimization objective from a fixed output string to matching the activations after a steering vector is added to the residual stream, so that the steering behavior can be reproduced by an input suffix alone, without requiring access to the model's internals at generation time. On four small open-weight models including Gemma-2-2B and Qwen3-1.7B, the German suffix almost completely reproduces the steering behavior and transfers well across prompts and across models; the anti-refusal (jailbreak) direction has a high success rate under the metric of avoiding refusal phrases, but only partially succeeds under the stricter HarmBench classifier, and cross-model transfer is also poor. The thesis also proposes a universal suffix trained across prompts, a model-translation method that midway uses a larger model to rewrite the suffix into readable text, and a soft-suffix variant as an upper bound for matching. The author states that the purpose of this equivalence study is defensive.

Why it was selectedThe study shows that the behavior of a steering vector can be reproduced by an input suffix, but the jailbreak direction has limited success rate and poor cross-model transfer.

Paper analysis

6 questions; open one to read the full answer

  1. What problem does this paper attempt to solve?

    The authors ask: can an input suffix reproduce the behavior of a steering vector, so that no access to model internals is needed at generation time.

    Core question: can behavior induced by a steering vector be reproduced by an input suffix alone, so that no access to model internals is needed at all at generation time.

    • Setting and motivation: activation steering adds a steering vector to the residual stream to guide model behavior; it is cheap and interpretable, but requires modifying hidden states at every generation step, which is infeasible when the model can only be accessed through a chat interface (per the authors).
    • Existing methods: GCG optimizes a discrete suffix from the input side, but its goal is to make the model output a fixed string (such as an affirmative prefix), not to target internal activations. The authors believe there is an unstudied equivalence relationship between the two.
    • Core observation: based on the linear representation hypothesis, the authors assume concepts correspond to directions in activation space; if a suffix produces activations at filler token positions that match "the activations after adding the steering vector," then that suffix should reproduce the same behavior.
    • Proposed in the paper: ActGCG (replacing GCG's loss with the squared error of activation matching), plus three extensions: universal suffix (shared across prompts), model translation (rewriting the suffix into readable text midway through optimization) and soft suffix (continuous embeddings, as an upper bound); tested on two concepts, German answering and anti-refusal (jailbreak).
    • Threat model:
      • Attacker goal: in the anti-refusal direction, make the model not refuse harmful requests and pass HarmBench judgment; in the German direction, make the model answer in German.
      • Knowledge: the optimization stage requires white-box access (building the steering vector, reading activations, running GCG); the paper does not explicitly label the threat model category using black-box/white-box terminology.
      • Capability: can only control the suffix of the input prompt; does not touch model internals at generation time.
      • Victim system: open-source small models (Gemma-2-2B-it, Qwen3-1.7B, Llama-3.2-3B, Qwen3-4B, etc.).
  2. What related research exists?

    Related work falls into two groups: steering vector construction, and GCG and its variants; SSR is the closest prior method.

    Steering vector construction methods

    • ActAdd (Turner et al., 2023): uses a single contrast pair, records activations at a certain layer and subtracts them to obtain a steering vector, which is added at one token position at generation time.
    • CAA (Rimsky et al., 2024): generalizes to multiple contrast pairs, taking the mean difference at the last token position of a single layer, added at every token position at generation time; this paper's steering vector follows this difference-of-means approach.
    • ITI (Li et al., 2023) and RepE (Zou et al., 2023): ITI trains linear probes in attention head space to select top-K heads for intervention, targeting truthfulness; RepE applies PCA to the difference vectors of each layer and takes the first principal component. The paper also mentions that SAE steering is harder to evaluate and has not shown a clear advantage (relaying others' discussion).

    GCG and its variants

    • GCG (Zou et al., 2023): optimizes a discrete suffix so that the model begins with a chosen affirmative string, using greedy search over top-k gradient coordinates; this paper keeps its discrete search and only changes the loss.
    • Steering-SSR (Winninger et al., 2025): the closest prior method, also optimizing a suffix to match activations after adding a steering vector, but using a per-layer cosine similarity loss. The authors say they could not reproduce its reported jailbreak success rate—after re-judging that paper's outputs with their own classifier, the resulting rate was clearly lower, which the authors attribute to the choice of jailbreak judge.
    • PEZ (Wen et al., 2023): gradient-based discrete optimization combining soft and hard, projecting continuous embeddings to the nearest vocabulary token each round and then backpropagating gradients.

    Other

    • Prompt translation (Li et al., 2024): the idea of rewriting unreadable suffixes into fluent text, borrowed in this paper for the midway rewriting step in the universal setting.
    • Baselines and benchmarks: comparison baselines are original GCG, SSR (two read-position variants) and soft suffix; data use the Dolly instruction dataset (German/English contrast pairs) and the harmful/harmless instruction set of Arditi et al. (2024); evaluation uses the HarmBench-Llama-2-13b classifier and the langdetect library.

    The authors position this paper as: changing GCG's objective from a fixed output string to the full target activations induced by a steering vector (L2 loss, multi-position multi-layer matching), and systematically testing the equivalence relationship between suffixes and steering vectors.

  3. How does the paper address this problem?

    ActGCG retains GCG's discrete search and replaces the loss with the squared error of the target activation of the steering vector.

    Overall approach: first build a steering vector using CAA-style difference-of-means, then replace GCG's cross-entropy loss with the squared error between "the activations from the suffix's forward pass" and "the target activation after adding the steering vector," and minimize it with standard GCG discrete search to obtain ActGCG.

    Steering vector construction

    • Method: CAA-style difference-of-means. The German direction uses German prompts from the Dolly dataset (positive) and their English counterparts (negative), read at the second-to-last token of the response (position −2); the refusal direction uses Arditi et al.'s harmful (positive)/harmless (negative) instruction sets, read at the last token of the prompt (−1), with Qwen3 read at −5 because the chat template inserts an empty reasoning scaffold.
    • Multiple layers: each model selects 4 layers toward the middle-to-later part (Table 3.1: Gemma-2-2B selects 10/14/18/22, Qwen3-1.7B selects 10/13/16/20, Llama-3.2-3B selects 12/16/20/24, Qwen3-4B selects 14/18/22/28), building one vector per layer.
    • Anti-refusal direction: takes the negative of the refusal vector (equivalent to negative α). The authors say the layers and positions were chosen after trying multiple combinations, and do not claim they are optimal.

    ActGCG objective

    • Target construction: for each target layer ℓ, add α·vℓ to every filler token position of the plain prompt (the fixed tokens in the chat template located after the user prompt and before the response), and read out the target activation tℓ,p.
    • Loss: L(s) = ∑ℓ ∈ L ∑p ∈ F | aℓ,p(x | s) − tℓ,p |2, i.e., the sum of squared errors between the suffix s's activations and the target activations across 4 layers and all filler positions; meaning: make the suffix's own forward pass reproduce the internal state after steering.
    • Why use filler positions: response tokens differ on each generation and have high variance; filler tokens exist for every prompt and are adjacent to the generation position, and multiple positions provide a stronger steering signal (per the authors).
    • Default parameters: 20-token suffix, initialized as "x x ... x", 250 steps, search width 512, top-k = 256, 1 token replaced per step, fixed seed 0; German α = 2.0, refusal α = 1.0.

    Universal suffix and model translation

    • Universal suffix: one suffix is jointly optimized over N training prompts, with the token replacement at each step determined by the sum of the gradients across prompts; generalization is evaluated on the training set and on new prompts, and the suffix is transferred to other models in the same family to evaluate cross-model transfer.
    • Model translation: following Li et al. (2024), first run half the steps (at step 125), call a larger proposer model (GPT-5.5 or Llama-3.1-8B) to rewrite the current suffix into fluent text of the same length, then continue optimizing from the rewritten result; an end_ask variant that rewrites only once at the end is also compared.

    Soft suffix and decision methods

    • Soft suffix: replace the 20 discrete tokens with continuous embeddings and directly optimize the same activation-matching loss with Adam (lr 0.05, 150 steps), as an upper bound on what discrete search can achieve.
    • German decision: counted as success only if the langdetect library outputs "de" and the text is not repetitively degenerate.
    • Refusal decision: the main metric is the jailbreak rate from the HarmBench-Llama-2-13b classifier; refusal phrase matching (phrase-based) is used as an auxiliary metric. The authors say they selected the Llama classifier after comparing gpt-5.5, claude-opus-4.8, and HarmBench-Mistral-7b-val-cls, on the grounds that the first two were difficult to obtain scores from and the Llama version is more commonly used in the literature.
  4. What experiments did the paper conduct?

    The German-direction discrete suffix nearly reproduces the steering vector; the anti-refusal direction has a high phrase-based score but a low HarmBench score.

    Experimental setup

    • Models tested: main models Gemma-2-2B-it, Qwen3-1.7B, Llama-3.2-3B, Qwen3-4B; transfer targets Gemma-1.1-2B, Gemma-2-9B, Qwen2.5-1.5B, Llama-3.2-1B.
    • Data: German uses the Dolly dataset, with 30 test prompts for per-prompt and 20 training + 30 test for universal; refusal uses Arditi et al.'s harmful test set, with 22 prompts in the breadth split (1 per category) and 60 in total in the depth split (5 training, 5 same-category test, 5 new-category test for each of 4 categories).
    • Baselines: original GCG (fixed target string), SSR (faithful single-token and filler variants), soft suffix (continuous-embedding upper bound), and a None baseline with no suffix.
    • Metrics and judge: German success rate uses langdetect (outputs "de" and is non-repetitive); the main refusal metric is the HarmBench-Llama-2-13b jailbreak rate, with phrase-based refusal-phrase avoidance as auxiliary.
    • Default settings: 20-token suffix, 250 steps, width 512, top-k 256, seed 0; German α = 2.0, refusal α = 1.0; all are single runs except the seed sweep (5 seeds).

    Main results

    Per-prompt comparison (Tables 4.2, 4.3, 4.7, 4.8; German n=30, refusal breadth n=22, %):

    Swipe horizontally to read the table

    ModelTaskOriginal GCGSSR (filler)ActGCG
    Gemma-2-2BGerman36.753.386.7
    Qwen3-1.7BGerman83.3not reported96.7
    Gemma-2-2Brefusal HarmBench13.622.727.3
    Gemma-2-2Brefusal phrase36.4100.090.9
    Qwen3-1.7Brefusal HarmBench50.0not reported40.9
    Qwen3-1.7Brefusal phrase100.0not reported90.9
    • The authors say that in the German direction ActGCG outperforms original GCG and both SSR variants on both models, and that reading at filler positions is better than a single token.
    • Refusal direction: ActGCG's HarmBench is higher than GCG and SSR on Gemma-2-2B, but lower than original GCG on Qwen3-1.7B; under the phrase metric ActGCG is slightly lower than the SSR variant. The authors say the activation objective is overall comparable to GCG's fixed-string objective, with no large improvement.
    • Upper bound of the objective itself: when the steering vector is added directly at filler positions, the German success rate is 90.00% (Table 4.1, Gemma-2-2B); in the refusal direction the HarmBench upper bound is 50.00% at α ∈ [1.0, 1.5] (Table 4.6), which the authors consider a possible reason for the overall low refusal success rate.

    Universal suffix and cross-model transfer

    • German (Table 4.4, α = 2.0, 20 training prompts): training set 90.0–95.0%, test set 73.3–96.7%; transfer to smaller models in the same family (Gemma-1.1-2B 65.0%, Qwen2.5-1.5B 95.0%, Llama-3.2-1B 95.0%) is better than transfer to larger models (Gemma-2-9B 5.0%, Qwen3-4B 60.0%).
    • Refusal (Tables 4.9, 4.10): breadth split training HarmBench 36.4–45.5%, test 31.8–59.1%; transfer drops markedly (e.g., Gemma-2-9B 0.0%, Llama-3.2-1B 4.5%). The authors say the suffix generalizes to unseen categories but cross-model transfer is poor; the phrase metric is about twice HarmBench.

    Model translation and soft suffix

    • German model translation (Table 4.5, rewritten at step 125): with GPT-5.5 as proposer, Gemma-2-2B training 95.0/test 96.7, reaching 100.0 when transferred to Gemma-2-9B (only 5.0 without translation, Table 4.4); Llama-3.2-3B training/test both 100.0. The authors say mid-optimization rewriting benefits German, especially cross-model transfer; the end_ask variant is worse (e.g., 0.0 with Llama-3.1-8B as proposer).
    • Refusal model translation (Tables 4.11, 4.12): overall worse than the universal version without translation; the authors say German may be easier to rewrite into a coherent meaning than the more complex refusal concept.
    • Soft suffix (Table 4.13, 30 German prompts): at α = 1.0, Gemma-2-2B 96.67% and Qwen3-1.7B 93.33%, higher than the discrete version, which the authors say shows that the discrete token space is a limitation; at α = 2.0, Qwen3-1.7B drops to 70.00%, which the authors attribute to the vector being too strong and causing empty outputs. Attempts to project soft tokens back to the nearest vocabulary token failed completely (0 German replies, Section 4.16).

    Other ablations and analyses

    • α sweep (Figures 4.1, 4.2): German α = 2.0 gives the highest success rate on Gemma-2-2B and Qwen3-1.7B, and the conclusion is unchanged across 5 seeds; the refusal suffix is slightly better at α = 0.5 than at 1.0, but the objective itself peaks at α ∈ [1.0, 1.5], so 1.0 is used by default.
    • Suffix length (Figure 4.3, Gemma-2-2B, 5 seeds): variance is low at length 20, and longer lengths yield little gain, so 20 is fixed.
    • More steps (Table 4.15, refusal breadth): 500 steps gives higher HarmBench than 250 steps (Gemma-2-2B 27.3→31.8, Qwen3-1.7B 40.9→54.5), but increases runtime, so 250 steps is still used.
    • Precision (Table 4.16, Gemma-2-2B, α = 1.0): fp32 70.00% is higher than bf16 53.33%, the gap is within seed variance, and bf16 is still used.
    • Reverse equivalence (Table 4.17): the steering vector recovered from the suffix can also reproduce the behavior, with the mean vector (80.00% at α = 2.0) better than the per-prompt vector (63.33%).
    • Prefix vs suffix (Table 4.18, Gemma-2-2B German): the best prefix is 66.67% (filler-only objective, α = 3.0), still lower than the suffix; objectives covering prompt positions are instead worse.
    • Initialization (Table 4.19): initializing with the readable instruction "Please answer in German" makes almost no difference from "x x ... x" (e.g., Qwen3-1.7B 96.67 vs 90.00).
    • Dataset construction variant (Table 4.14, α = 1.0): fixing the English prompt and comparing only the answer language gives no consistent improvement (Qwen3-1.7B 80.00→83.33, Gemma-2-2B unchanged at 53.33).
  5. What are some points that can be further explored?

    The authors list limitations including concept coverage, scale, discrete space, and the judge, and propose testing more concepts and improving the loss and vector construction.

    Limitations and future directions noted by the authors

    • Concept coverage (Section 5.1): the success rate heavily depends on the concept being steered; only German and refusal were tested, so performance on other concepts cannot be inferred.
    • Scale (Section 5.1): the evaluation scale is small (30 prompts for German, 22–60 for refusal), and most runs use a single fixed seed, so scores may vary with prompt or seed.
    • Discrete space limitation (Section 5.1): discrete token search limits the precision of matching the target activation; the lower loss of the soft suffix constitutes an upper bound, and the gap between them reflects the success-rate loss caused by the discrete space.
    • Limited by the steering vector (Section 5.1): the suffix can only reproduce behavior that the steering vector itself can induce; and matching activations is not a guarantee of behavior—a suffix can be close to the target activation yet still fail to jailbreak.
    • Imperfect judge (Section 5.1): safety measurement heavily depends on the judge; phrase-based metrics are not a good proxy for safety (avoiding refusal phrases does not equal giving harmful content); langdetect occasionally misjudges short German replies containing English names.
    • Future directions (Section 5.3): test more concepts; try more robust losses beyond squared L2; find better steering vector construction methods than difference-of-means; test the equivalence relation with non-steering-vector targets such as activations from other models.

    Experimental coverage

    • 8 models were tested: 4 main models (Gemma-2-2B-it, Qwen3-1.7B, Llama-3.2-3B, Qwen3-4B) plus 4 transfer targets, all open-source small models.
    • Only 2 concepts: German answers and anti-refusal; sample sizes are 30 for German (per-prompt) / 20+30 (universal), and 22 for refusal breadth and 60 for depth.
    • Except for the α sweep and suffix length experiments on Gemma-2-2B, which ran 5 seeds, all others are single runs with a single seed 0; the paper reports no confidence intervals or statistical tests.
    • The refusal judge uses only one classifier, HarmBench-Llama-2-13b (the authors say they selected it after comparing gpt-5.5, claude-opus-4.8, and HarmBench-Mistral-7b-val-cls); the paper reports no agreement between the judge and human judgment.
    • The optimization budget is fixed at 250 steps and a 20-token suffix (variants with 500 steps and lengths 5–50 were also tested); the paper reports no specific time or query-count numbers per optimization.
  6. Summarize the main content of the paper

    ActGCG reproduces steering vector behavior with discrete suffixes: near success for German, anti-refusal weaker under strict evaluation.

    A bachelor's thesis on the equivalence between activation steering and input attacks: the authors ask whether the behavior induced by a steering vector can be reproduced by an input suffix alone, so that generation requires no access to the model's internals.

    • Method: ActGCG keeps GCG's discrete token search and replaces the cross-entropy loss with the squared error between the suffix activations and the "target activation after adding the steering vector" at filler token positions across 4 layers; there are also three extensions: universal suffix (joint optimization over multiple prompts), model translation (rewritten into readable text by GPT-5.5 or Llama-3.1-8B at step 125), and soft suffix (continuous-embedding upper bound).
    • German direction: per-prompt success rates are 86.7% for Gemma-2-2B and 96.7% for Qwen3-1.7B (30 prompts, Table 4.2), close to the 90.00% upper bound of the steering vector itself (Table 4.1), and better than original GCG (36.7/83.3) and SSR (up to 53.3).
    • Anti-refusal direction: on Gemma-2-2B the phrase-based success rate is 90.9%, but the HarmBench jailbreak rate is only 27.3% (n=22, Table 4.7); on Qwen3-1.7B HarmBench is 40.9%, lower than original GCG's 50.0%. The authors say the substring metric is about twice HarmBench, and the equivalence is weaker on this concept.
    • Transfer: the German universal suffix transfers well to smaller models in the same family (95.0% for both Qwen2.5-1.5B and Llama-3.2-1B) and poorly to larger models (5.0% for Gemma-2-9B); after model translation, German transfer improves (Gemma-2-9B reaches 100.0%), while refusal gets worse overall.
    • Reverse equivalence: the steering vector recovered from the suffix can also reproduce the behavior (80.00% with the mean vector at α = 2.0, Table 4.17), which the authors take as support for the equivalence between suffix and steering vector.
    • Authors' conclusion: the behavior of a steering vector can be achieved from the input side, with no model-internal access during generation, but the loss depends on the concept; the authors say the practical attack value is limited (jailbreak rates are not high, cross-model transfer is poor), and position the work as defense motivation—groundwork for defending against inputs that "reproduce malicious concept vectors."
Read originalarxiv.org