Are aligned neural networks still aligned under adversarial inputs?

Core question: whether language models that have undergone alignment training remain aligned when facing adversarial users with worst-case constructed inputs (the authors call this adversarial alignment).

• Problem setting: alignment techniques (instruction tuning, RLHF) make models "helpful and harmless", but humans can already induce harmful outputs through "social engineering"-style jailbreaks; the authors believe this phenomenon is difficult to study scientifically and needs to be systematically evaluated using the adversarial example framework.
• Shortcomings of existing methods: the authors claim existing NLP optimization attacks (such as ARCA, GBDA) have low success rates in chatbot settings, but this cannot distinguish between the two explanations of "the model is truly robust" and "the attack is too weak", so attack failure should not be taken as evidence that the model remains aligned under adversarial inputs.
• Core observation: multimodal models accept image inputs in the continuous domain, and image adversarial examples are much easier to construct than discrete text adversarial examples; the authors conjecture that improved NLP attacks might achieve the same adversarial control on text-only models (the authors explicitly label this as a conjecture).
• Paper contributions: systematically evaluate the effectiveness of existing NLP attacks on aligned chat models; construct test cases that guarantee a solution exists to test attack strength; carry out image adversarial attacks on three open-source multimodal models.
• Threat model (Section 3.2):
  • Objective: trigger the model to output toxic content, formalized as isToxic(Gen(X)) being true; isToxic uses simple substring matching (whether the output contains the selected harmful word).
  • Knowledge: white-box, the attacker can access the entire model (Appendix A).
  • Capability: the malicious user can only control tokens within their own message (at most 30 in the evaluation), cannot modify the chat history and special tokens such as [USER]:/[AGENT]:; in the multimodal setting can provide adversarial images.
  • Victim systems: aligned chat models (GPT-2, LLaMA, Vicuna) and open-source multimodal models (Mini GPT-4, LLaVA, LLaMA Adapter); the authors claim the attack may be practical in both the malicious user and malicious third party (prompt injection) scenarios.

Alignment training

• Bai et al. (2022), Ouyang et al. (2022), Christiano et al. (2023)—use RLHF and instruction tuning to make models "helpful and harmless"; these are the sources of the alignment techniques evaluated in this paper.
• Wei et al. (2022a)—instruction tuning, fine-tuning models on tasks with instruction descriptions.

Text adversarial examples and jailbreaks

• Jones et al. (2023)—the ARCA attack, which uses discrete optimization to find prompts that make the model output a target string, and has made models output the names of U.S. senators and toxic words. The paper notes that its threat model is stronger: the attacker controls the entire prompt or the text immediately preceding the generation position, which does not match chatbot deployment; this paper's setting is more general.
• Wallace et al. (2019), Ebrahimi et al. (2017), Alzantot et al. (2018), Jia and Liang (2017)—universal adversarial triggers, HotFlip, and other text adversarial attacks; the paper states that ARCA generalizes and outperforms these works, so they are not re-evaluated.
• Guo et al. (2021)—GBDA, a gradient-based text adversarial attack, one of this paper's baselines.
• Greshake et al. (2023)—prompt injection attacks, corresponding to the malicious third-party scenario in this paper's threat model.

Reliability of attack evaluation

• Zimmermann et al. (2022)—proposes a framework to distinguish "the defense is truly robust" from "the attack is too weak": first construct test cases known to contain adversarial examples, then test whether the attack can find them.
• Lucas et al. (2023)—this paper's specific method for constructing test cases follows this work.

Multimodal models

• Mini GPT-4 (Zhu et al., 2023), LLaVA (Liu et al., 2023), LLaMA Adapter (Gao et al., 2023)—open-source image-text models that use a CLIP-like vision encoder plus a projection to feed image embeddings into the language model; these are the attack targets of this paper.
• GPT-4 (OpenAI, 2023), Flamingo (Alayrac et al., 2022), Gemini (Pichai, 2023)—closed-source multimodal models; the paper states that because they were not publicly accessible, it tested open-source implementations instead and did not run these models itself.

Baselines and benchmarks: the baseline attacks are ARCA and GBDA; benign conversations come from the Open Assistant dataset, and toxic target phrases are taken from Jones et al. (2023).

The authors position this paper as unifying the two directions of adversarial examples and AI alignment, and emphasize that the purpose is not to propose better attacks but to lay the groundwork for defense evaluation.

The overall approach has three steps: evaluate existing NLP attacks under real chat template constraints; use brute-force search to construct test cases that "guarantee a solution exists" to determine whether attack failure is due to model robustness or the attack being too weak; use the continuous domain of images to perform gradient attacks on multimodal alignment models. The attack objective is uniformly harmful-prefix: make the model's reply begin with a harmful prefix.

Attack objective and success determination

• Harmful prefix objective: rather than directly optimizing isToxic(S), optimize the surrogate objective S..j = t (j ≪ |S|), i.e., make the first j generated tokens equal the harmful string t, requiring only one forward pass. The authors claim that as long as the model begins with harmful content, it will continue to output harmful text afterward without additional adversarial control.
• Success determination: isToxic is substring matching—if the generation contains the selected harmful word, it is judged toxic; the experiment sets temperature to 0 to make the model output deterministically.

NLP attack evaluation setup (Section 4)

• Chat template constraints: the attacker can only control the [USER]: message content; special tokens such as [AGENT]: cannot be modified; formalized as making Gen(Ppre ||X|| Ppost) toxic.
• Data construction: take benign conversations from Open Assistant, truncate to a random K ∈ [1..N] turns, then append a toxic target; the attack target is 1–3 token toxic text, taken from a subset of Jones et al. (2023).
• Attack configuration: the attacker can modify at most 30 tokens; two positions, Distant (adversarial tokens before the question) and Nearby (after the question); evaluate ARCA and GBDA.

Attack strength test (Section 5)

• Test case construction: sample prefixes pj from Wikipedia, enumerate (or sample 1,000,000) N-token sequences si, find pj for which the output set {qi} has small but positive entropy, and set the attack target to the least likely token qi among them; prevalence is fixed at 10^-6.
• Determination method: brute-force search has guaranteed a solution exists, so the attack passes if it finds any input such that Gen(p') = q; any failure indicates the attack is flawed. Construct 8 test sets of different difficulty and average them; the number of attacker-controllable tokens is 2, 5, 10, 20, with additional token budgets of 1×/2×/5×/10×.
• Why GPT-2: brute-force search is costly, so only GPT-2 is used for construction; the authors claim GPT-2 is easier to attack, and if the attack fails on it, it is unlikely to succeed on harder-to-attack aligned models.

Multimodal image attack (Section 6)

• Differentiable pipeline: build an end-to-end differentiable implementation from image pixels to language model output logits, use PGD (Madry et al., 2017) to optimize the adversarial image, maximizing the likelihood that the model begins with the specified harmful reply; when the target exceeds 1 token, use teacher forcing to optimize the sum of cross-entropy over each target token.
• Attack configuration: the initial image is a random image sampled uniformly per pixel; ϵ is taken arbitrarily large, at most 500 steps or stop on success; step size 0.2; report the final ℓ2 perturbation.

Experimental setup

• Models tested: text models GPT-2 (no chat training, no alignment), LLaMA (can chat, no explicit alignment), Vicuna (aligned via fine-tuning); multimodal models Mini GPT-4 (both Instruct and RLHF versions), LLaVA, LLaMA Adapter.
• Data: benign conversations come from Open Assistant, truncated to a random K ∈ [1..N] turns; toxic targets are taken from Jones et al. (2023), 1–3 tokens; prefixes for attack strength tests come from Wikipedia.
• Baseline attacks: ARCA (Jones et al., 2023) and GBDA (Guo et al., 2021); the control for attack strength tests is brute-force search.
• Metrics: attack success rate (isToxic substring matching), test case pass rate, average ℓ2 perturbation of multimodal attacks.
• Attack budget: text attacks use at most 30 adversarial tokens; test cases have 2/5/10/20 controllable tokens, with an additional 1×–10× extra budget; image attacks use at most 500 PGD steps.
• Other: temperature is set to 0; the paper does not report the number of repetitions or sample size for each experiment group (Table 1 gives ± error without stating its source).

Main results

Table 1: Text attack success rates (at most 30 adversarial tokens)

Swipe horizontally to read the table

Model	Alignment	ARCA Distant	GBDA Distant	ARCA Nearby	GBDA Nearby
GPT-2	None	67% ± 4%	12% ± 3%	84% ± 3%	16% ± 3%
LLaMA	None	2% ± 1%	1% ± 1%	61% ± 6%	10% ± 1%
Vicuna	Instruct	6% ± 2%	0% ± 1%	8% ± 2%	1% ± 1%

Table 3: Multimodal image attacks (harmful-prefix, PGD)

Model	Attack success rate	Average ℓ2 perturbation
LLaMA Adapter	100%	3.91 ± 0.36
Mini GPT-4 (Instruct)	100%	2.51 ± 1.45
Mini GPT-4 (RLHF)	100%	2.71 ± 2.12
LLaVA	100%	0.86 ± 0.17

• The authors state that alignment fine-tuning is an effective defense against existing white-box NLP attacks: on Vicuna, the success rate of each attack does not exceed 8% ± 2%.
• The authors state that multimodal attacks succeed under all prompts, with small perturbations and little variation across prompts; LLaVA requires about 1/10 the perturbation of the other models, which the authors state shows that implementation details affect vulnerability; the paper does not explain why.

Attack strength tests (Table 2)

• What was tested: on GPT-2, brute-force search was used to construct test cases guaranteed to have a solution (prevalence 10^-6), to test whether ARCA and GBDA can find adversarial inputs known to exist.
• Results: brute-force search has a 100.0% pass rate at all budgets; ARCA at 1×/2×/5×/10× budgets is 11.1%, 14.6%, 25.8%, 30.6% respectively; GBDA is 3.1%, 6.2%, 8.8%, 9.5%.
• Authors' interpretation: ARCA fails nearly 90% of tests at the 1× budget, and still fails most tests even with 10× extra tokens; the authors state that this shows existing attacks are insufficient to evaluate the true robustness of aligned models, rather than that these attacks themselves are wrong.

Qualitative evaluation

• Qualitative toxicity check (Section 4.4): manual inspection of some generations for each setting in Table 1 found that most were indeed toxic; but in fewer than 10 cases, Vicuna first output a harmful phrase and then self-corrected (e.g., stating "this is inappropriate"), which the authors state exposes the limitation of substring-matching isToxic.
• Qualitative multimodal examples (Section 6.4, Appendices C/D/E): under clean inputs the models refuse or output harmless content; under adversarial images they output harmful content; the authors note that the response quality on "how to get away with murder" is poor, and believe this may be due to insufficient model capability, and that better models might give more actionable advice.

Other ablations and analyses

• Adversarial token position: Nearby is generally higher than Distant (Table 1, e.g., on GPT-2, ARCA rises from 67% ± 4% to 84% ± 3%).
• Attack budget: test case pass rate rises with the multiple of extra tokens (Table 2, ARCA rises from 11.1% to 30.6%).
• Differences in vulnerability across models: LLaVA's average ℓ2 perturbation is 0.86 ± 0.17, while the other models are 2.51–3.91 (Table 3).

Limitations and future directions noted by the authors

• Limitation of toxicity definition (Section 4.4): the substring-matching isToxic misjudged in a few cases (fewer than 10 cases); the authors state that future work needs to carefully consider different toxicity definitions.
• Stronger NLP attacks (Section 7): the authors hope that stronger NLP attacks can comprehensively evaluate the robustness of aligned LLMs, and state that such attacks should at least pass this paper's test cases to be considered reliable.
• Robustness of text models remains open (Sections 1, 7): the true adversarial robustness of alignment techniques requires stronger attacks to answer; the authors conjecture that improved NLP optimization attacks may induce aligned models to output harmful content (explicitly labeled as conjecture).
• Differences in vulnerability across models (Section 7): design decisions make attack difficulty differ by up to 10×; the authors state that understanding the sources of vulnerability is an important future direction.
• New modalities (Section 7): the authors state that future models are likely to add modalities such as audio, which may introduce new attack surfaces.
• Attack-defense transfer (Section 7): whether insights on attack and defense from the adversarial machine learning literature can transfer to this field is stated by the authors as an important open question.

Scope of experiments

• Text attack evaluation covers three models, GPT-2, LLaMA, and Vicuna, with baselines ARCA and GBDA (Table 1).
• Attack strength tests are constructed only on GPT-2 (Section 5.1, due to the cost of brute-force search).
• Multimodal attacks cover three open-source models, Mini GPT-4, LLaVA, and LLaMA Adapter; GPT-4 and Gemini were not tested because they were not publicly accessible (Section 6.2).
• All attacks are in a white-box setting; Appendix A states that the attacks do not directly apply to production models such as GPT-4.
• The paper does not report the sample size or number of repetitions for each experiment; the denominator for the qualitative toxicity check is not given (only stating "fewer than 10 cases" were anomalous).

An attack paper on "adversarial alignment," asking whether models after alignment training remain aligned under worst-case inputs.

• Problem: existing NLP attacks fail on aligned chat models, and it is impossible to distinguish "the model is truly robust" from "the attack is too weak"; the authors state that attack failure should not be taken as evidence that the model remains aligned.
• Method: evaluate ARCA and GBDA under chat template constraints; use brute-force search to construct test cases guaranteed to have a solution to test attack strength; for multimodal models, use PGD to perform harmful-prefix attacks in the continuous image domain (white-box, temperature 0, toxicity determined by substring matching).
• Result one: on Vicuna, the success rate of each NLP attack is at most 8% ± 2% (Table 1); the authors state that alignment fine-tuning is an effective defense against existing white-box NLP attacks.
• Result two: on GPT-2, among tests guaranteed to have a solution, brute-force search passes 100.0%, ARCA reaches only 30.6% even with 10× extra tokens, and GBDA 9.5% (Table 2); the authors state that existing attacks are insufficient to evaluate true robustness.
• Result three: on Mini GPT-4, LLaVA, and LLaMA Adapter, image attacks have a 100% success rate; LLaVA's required ℓ2 perturbation (0.86 ± 0.17) is about 1/10 that of the other models (Table 3).
• Authors' conclusion: multimodal models are easily controlled by continuous-domain attacks; the authors conjecture that improved NLP attacks may achieve the same effect on text-only aligned models (explicitly a conjecture), and call for stronger attacks to support defense evaluation.