Persona Attack: Jailbreaking Large Language Models via Incremental Memory Injection

How to use user instructions written step by step into dialogue memory to make the model prioritize these instructions over internal safety alignment.

• Scenario: The authors state that chat agents are already used in consumer and professional scenarios, and may also be used to enhance illegal activities. Developers use safety policies, safety training, and filtering to prevent misuse, but content moderation still struggles to cover complex boundaries and is vulnerable to specific prompts.
• Existing shortcomings: The authors state that traditional jailbreaks are mostly single-shot prompt injection, ignoring that models remember dialogue flow and user instructions. Directly asking harmful content usually gets no answer; after stepwise instruction injection, dangerous responses appear (Figure 1).
• Hypothesis: The authors believe that accumulated user instructions in memory are given higher weight than safety goals, so user instructions take precedence over safety policies.
• Approach: Propose Persona Attack (incremental memory injection). Use four prompts to complete imprinting and attack, formulate the combination as a search to maximize ASR, and perform pruning.
• Threat model:
  • Goal: Bypass safety mechanisms to make the model produce harmful content (what the paper calls jailbreak).
  • Knowledge: The paper states GPT-4o is black-box, internal alignment is not public; GCG cannot be optimized on it (Figure 4, 5.1).
  • Capability: The attacker can submit I1, I2, I3 and the harmful question Qh sequentially or at once, thereby influencing dialogue history or internal state.
  • Victim system: LLMs with dialogue memory. Experimental targets are GPT-4o on the Responses API, Llama-3.2-3B-Instruct on LangChain, and ChatGPT desktop application, Claude 3.7 sonnet via Perplexity pro, and Grok 3 Beta.

Section 2 categorizes related jailbreaks into three types according to [7]. Differences are only written where the paper explicitly compares.

• Generative jailbreaks: Use dynamic generation and iterative feedback to modify prompts in real time [8]. The authors state that AutoDAN and GCG are white-box, using gradients to optimize suffixes to induce harmful outputs [9][10]; GPTFuzzer is black-box, using user prompts as seeds for transformations [11]. The authors cite [7] that they can be consistently applied to various harmful questions and are hard to detect, but implementation is complex and often requires model internals. The authors state that this paper also aims to exploit vulnerabilities in internal workings, but uses multi-turn memory rather than a single prompt.
• Template-based jailbreaks: Use prompt patterns or role-playing to induce harmful outputs. DAN makes the model act without constraints, DeepInception uses fictional narratives to exploit character capabilities [6][12]. The authors state that these do not require separate optimization, but are easily detected when prompts are repeated, and tokens are often longer. The authors state that this paper also uses predefined instructions, which can reduce tokens relative to some complex single-turn templates, and the final query has fewer malicious patterns; the paper does not report token counts.
• Exploiting training gaps: Exploit inputs or scenarios where safety alignment does not generalize. MSJ places hundreds of harmful Q&A pairs in the prompt to weaken refusal [13]; the authors also mention low-resource languages, code formats, and special tokens. The authors state that this paper does not rely on long prompts, but uses multi-turn memory injection to gradually weaken defenses, producing a similar effect of breaking safety priorities; the paper does not provide comparative numbers with MSJ.
• Baselines and data: Figure 4 compares GCG [15], Jailbroken [21], Parameters [22]. The caption states that GPT-4o uses the Responses API and Llama uses LangChain, both state-based, and GCG cannot be optimized on the black-box GPT-4o. The main experiment uses 60 prompts selected from AdvBench [20]; the appendix also uses 520 AdvBench prompts [15].

The authors state at the end of Section 2 that this paper differs from single-turn or static prompts by exploiting interactive memory; they call this path stealthy and efficient. The paper does not report query counts or time costs.

Persona Attack 利用模型处理顺序提示时的记忆机制：先把指令印进记忆，再让有害问题沿已记住的回答框架生成。作者称用四条提示即可实施，不同于其所说的其他复杂越狱。

指令印刻

• 三条指令：集合 I={I1,I2,I3}。I1 为模仿，让模型预测其他语言模型的回答；I2 为假设注入，把任务说成按格式作答，并规定四种情形：Jailbreak Fail、Partial Fail、Partial Success、Success；I3 为去掩码，要求完整输出、不对特定词掩码（Figure 2，3.1）。附录 B 给出完整提示词。
• 作者的机制说法：I1 与 I2 表面上像破冰。与固定轨迹结合后，模型应记住符合该格式的问题要归入四种情形之一，随后插入的问题被套进这些框架。作者称模型保持对指令的顺从，未意识到回答在变有害，从而把有害问题当成合法的模拟回答。
• 另一因素：作者认为四种情形里已有部分有害的情形，成功情形会导向有害输出；模型为贴合目标情形而逐步选择更有害的 token，并倾向于不识别单个 token 在整体中的有害程度（3.4）。

攻击步骤与排列

• 触发：Qh 被放进包装层，以便模型为维持上下文而调用已存指令（3.2）。此处不复述包装句。
• 基本排列：Sequential 1 依次为 I1、I2、I3、Qh；Sequential 2 为 I1、I2、Qh、I3；Once 把四段 concat 后一次输入。Qh 也可插在 I 的元素之间。
• 其他排列：正文称 Additional 1、Additional 2 是重排后的 sequential 变体。附录 Table 4 给出 Additional 3 至 6 的拼接顺序，并写明假定 state-based memory。

组合搜索

作者将攻击写成在合法组合上最大化 ASR：

max_{X ∈ C(I,Qh)} ASR(LLM(X))

C(I,Qh) 是 I1、I2、I3 与 Qh 的合法组合空间，X 是其中一种输入组成。作者排除被判断为不太可能成功的组合，例如 Q 先输入，只实验其余配置。论文未给出剪枝的形式规则，也未穷尽该空间（3.3）。

作者给出的权重解释

作者认为成功来自安全对齐与记忆中用户指令的失衡，并把总目标写成两个分布的加权和：

P = α · P_instruction + β · P_safety, α > β

前者对应遵循记忆中的用户指令，后者对应安全训练。作者称当 α 大于 β 时，模型把指令遵循当作更主导的偏好，因而可能在违反安全约束时仍按指令生成。论文未报告 α、β 的估计（3.4）。

两种记忆

• Manual：把先前对话成对拼接后贴在新提示前；在不超过上下文窗口的范围内迭代，作者称该窗口可视为记忆上限（Figure 3）。各轮回答条件于已有指令与回答，最后在拼好的 Imem 上回答 Qh。
• State-based：维护内部状态。s0 初始化为 0；每轮 rt 为在上一状态条件下对 xt 的回答，st 由更新函数 M 根据上一状态、当前输入和回答生成。四步输入为 I1、I2、I3、Qh。作者认为历史如何被处理、又如何交给模型，会实质影响 ASR（3.5）。
• 实现对应：4.1 写 GPT-4o 用 Responses API、Llama-3.2-3B-Instruct 用 LangChain 添加记忆；Figure 4 图注称这两种环境都是 state-based。

Experimental Setup

• Models: Main tests are GPT-4o and Llama-3.2-3B-Instruct. The former is described as capable of processing text, images, and audio; the latter is a multilingual instruction model supporting a 128k token context. Real services are ChatGPT-4o (desktop app), Claude 3.7 sonnet, and Grok 3 Beta (the latter two via Perplexity pro). Appendix Tables 4 and 5 list the model as gpt-4o-2024-11-20.
• Data: Tables 1 and 2 use 60 prompts selected from AdvBench [20]. Appendix A states that these 60 prompts follow [20], adding some AdvBench to the 40 from [29], then divided into 6 topics with 10 per topic. The appendix also evaluates 520 AdvBench prompts [15].
• Memory and baselines: Compares manual and state-based. Figure 4 also compares GCG, Jailbroken, Parameters; the caption states that GCG cannot be optimized on the black-box GPT-4o.
• ASR / FAR: Non-refusals count toward ASR; only user prompts that are detailed and contain harmful sentences count toward FAR. Appendix D adds: FAR also requires following the injected instructions; if key information is masked, it does not count.
• Judgment: The paper does not specify who judged the 60 prompts, nor does it report the number of repetitions. The 520 prompts, being too long for manual checking, were instead sampled from successful cases; if the answer contains refusal phrases, it is judged a failure, and the authors call this result FAR (Appendix A, Figure 6).

Main Results

According to Table 2. The text uses these values as percentages for ASR and FAR, with data from 60 prompts.

Swipe horizontally to read the table

Model and Memory	Once ASR	Sequential 1 ASR	Once FAR	Sequential 1 FAR
GPT-4o Manual	75	88.33	63.33	83.33
GPT-4o State-based	75	95	63.33	88.33
Llama-3.2-3B-Instruct Manual	20	5	18.3	5
Llama-3.2-3B-Instruct State-based	20	35	18.3	20

• The authors state that writing instructions stepwise makes the model more likely to accept harmful context than putting them in at once; they also believe the state-based update function contains some processing, hence the difference from the manual sequential results. The text uses an approximate phrasing for the GPT-4o manual ASR in that case, differing from the decimal notation in the table.
• For Llama, under Manual, Once is higher than Sequential. The authors call this counterintuitive and suggest that multi-turn accumulation approaches MSJ, and that the model's training criteria target many-shot jailbreak, with SFT data containing unsafe behavior examples and safe behavior examples, thus defending against long contexts; Once is a single prompt with a different structure, thereby bypassing that defense logic.
• The authors state that in successful conversations, the model gradually treats the user instructions in memory as commands above safety alignment. In Table 1, Jailbroken and Parameters have ASR and FAR both 0, and the table title does not specify the model. In the text for Figure 4, Persona has a minimum success rate of 63.3%, and the two existing methods have no successful cases; the sentence does not name which two. GCG is described as not optimizable on this black-box, and Table 1 does not include its ASR.

Instruction Combinations

• Table 1 (table title does not specify the model) reports Once ASR 75.0, FAR 63.3, Sequential 1 as 95.0 and 88.3, Additional 1 as 91.7 and 86.7, Additional 2 as 86.7 and 76.6. The text states the latter two are sequential variants with reordered prompts, but does not give the order in that table. The Sequential 1 and Once numbers match GPT-4o State-based Sequential 1 and GPT-4o Once rows in Table 2, respectively; the paper does not specify the model for Table 1 based on this.
• Table 4, under gpt-4o-2024-11-20, 60 prompts, assuming state-based: Sequential 1 is 57/60 (ASR 95.0) and 53/60 (FAR 88.3); Sequential 2 and Additional 4 are both 57/60 and 54/60 (95.0, 90.0); Additional 3 is 54/60 and 49/60 (90.0, 81.7); Additional 5 is 50/60 and 40/60 (83.3, 66.7); Additional 6 is 50/60 and 47/60 (83.3, 78.3); Once is 44/60 and 38/60 (73.3, 63.3).
• The highest FAR in that table is 90.0, appearing in Additional 4 and Sequential 2. Once's ASR of 73.3 does not match Table 1's 75.0, while FAR is 63.3 in both. The paper does not explain this discrepancy.

Topics and AdvBench 520

• Table 5 (gpt-4o-2024-11-20, 10 questions per topic): Harmful Content ASR 90, FAR 80; Adult Content 90 and 70; Illegal Activity 90 and 90; Spreading Fake Information-Fraud and Unlicensed Activity both 100 and 100; Prejudice Privacy 100 and 90. The authors state that some topics may be more vulnerable, possibly due to differences in topic sensitivity, risk compensation, or alignment-stage constraints.
• AdvBench 520, gpt-4o, refusal-phrase rule: FAR is 84.61%. The text states 440 are judged successful and 80 fail due to refusal phrases (Figure 6). The authors state that light red indicates fully accepted harmful answers, and light green indicates refusal or partial refusal.

Real Services

• The text first uses "plan to" to state experiments on ChatGPT-4o, Claude 3.7 sonnet, and Grok 3 Beta, then Figure 5 reports sequential 1 results on the same 60 prompts. The authors state that they can bypass the latest LLMs in the services, a real threat rather than just theoretical; they also state that Claude is relatively robust but still has critical FAR. The converted text does not include the bar heights or labeled numbers from that figure, so ASR/FAR for each service is not written.
• Appendix E gives sequential 1 examples for the above services, distinguishing complete failure from partial failure. The authors state that the risk of the question greatly affects the weight β on Psafety: when very risky, β is greater than α; when relatively not risky, β is less than α. The paper does not measure β.

Other Ablations and Analyses

• The abstract states up to 95% under specific instruction configurations; the introduction states up to 95% higher than single-prompt methods (0%); the conclusion states the highest ASR on gpt-4o is 95%.
• The conclusion states that Llama-3.2-3B-Instruct's ASR ranges from 5% to 35% depending on whether memory is used, matching the Manual and State-based cells for Sequential 1 in Table 2.
• The paper does not report sampling temperature, number of repetitions, number of queries, or token usage.
• Appendix D: complete failure is not following the injected instructions and refusing; partial failure is following the framework but refusing in successful cases.

Limitations and future directions noted by the authors

• Black-box not directly analyzable: Taking gpt-4o as an example, internal alignment is not public. The authors state that only the bypass can be observed from outputs, making it difficult to directly analyze how safety training and internal alignment are disrupted, thus limiting the interpretation and generalization of black-box results (5.1).
• Opacity hinders designing defenses: The authors state that if the alignment process or rules were public, it would be possible to detect and remove anomalous instructions in memory, or add a filtering layer to manage dialogue context; opacity may also affect future research extensions (5.1).
• Related to accessibility of memory updates: The authors state that the more an attacker can persistently influence the dialogue history, the higher the likelihood that safety mechanisms are disabled; if the model provides functionality to update internal state through interaction, this attack would demonstrate an effect of bypassing safety alignment (5.1).
• Suggested mitigations by the authors: Identify and ignore inappropriate instructions in dialogue memory, or periodically initialize long-context to prevent dangerous instructions from accumulating gradually (5.2).
• Usability: The authors state that preventing such attacks without compromising usability is technically complex and involves ethical considerations, beyond the scope of this study (5.2).
• Directions in the conclusion: Continuous safety alignment based on model memory is needed; as multimodal models increase and context windows lengthen, subsequent research is needed to strengthen safety mechanisms to prevent this convenience from being used for attacks (6).

Experimental coverage

• Main test models are GPT-4o and Llama-3.2-3B-Instruct; real services include ChatGPT-4o, Claude 3.7 sonnet, Grok 3 Beta (4.1, Figure 5). The paper describes GPT-4o as capable of processing text, images, and audio, with the specified attack input being text prompts.
• Table 2 compares Once vs. Sequential 1 and manual vs. state-based, with data from 60 items. The appendix provides 7 combinations in Table 4 and 6 topics in Table 5 on gpt-4o-2024-11-20; the combination search excluded permutations the authors judged unlikely, such as Q input first (3.3).
• The phrase rule results for 520 AdvBench items specify the target as gpt-4o (Figure 6). Figure 5 is for sequential 1, 60 items; the main text does not provide ASR or FAR for each commercial service.
• Baseline attacks include Jailbroken and Parameters with ASR and FAR both 0 in Table 1 (table title does not specify the model), and GCG which the figure caption states cannot be optimized on black-box GPT-4o (Figure 4).
• The paper does not report the judge, number of repetitions, number of queries, or agreement with human judgment for the main experiments; the 520 items use refusal phrase rules (4.1, Appendix A). Section 5.2 states that defenses that do not compromise usability are beyond the scope of this study, and the experimental section does not report defense evaluation numbers.

Persona Attack performs jailbreaks using incremental memory injection, examining whether user instructions in conversational memory are prioritized over safety alignment.

• Problem: A single prompt injection does not exploit the model's ability to remember conversations and user instructions. The authors argue that instructions written in steps are prioritized over safety policies.
• Method: First inject imitation, hypotheses for four answer scenarios, and unmasking requirements, then submit a harmful question placed in the wrapper layer. Compare one-time concatenation with multi-turn sequential, and manual with state-based memory. ASR counts non-refusals, FAR counts detailed harmful instructions, and the appendix requires following injected instructions.
• Main numbers: In Table 2, GPT-4o state-based Sequential 1 has ASR 95 and FAR 88.33, while Once has ASR 75. In the same table, Llama-3.2-3B-Instruct Sequential 1 has manual memory ASR 5 and state memory ASR 35. In Table 1, Jailbroken and Parameters have ASR and FAR 0, and the table title does not specify the model.
• Other numbers: The appendix on gpt-4o, over 520 AdvBench items, counts by refusal phrases and obtains FAR 84.61% (440 judged successful, 80 judged failed). Table 5 on gpt-4o-2024-11-20 shows Spreading Fake Information-Fraud and Unlicensed Activity both have ASR and FAR 100 (10 questions each).
• Authors' conclusion: Long-term memory of context increases convenience but also lets attackers persistently implant instructions and bypass ethical limits. The authors state the highest ASR on gpt-4o is 95%, varying with memory implementation; memory-based safety alignment is needed. The introduction also states up to 95% higher than single-prompt methods (0%), alongside the abstract and conclusion's "reaching 95%", and the paper does not unify this wording.