EquaCode: Multi-Strategy Jailbreak via Equation Solving and Code Completion

Existing jailbreaks mainly use a single strategy in natural language, and the authors believe this is insufficient to evaluate the robustness of aligned LLMs.

• Scenario: The authors state that LLMs are used for question answering, translation, and code generation, but their generative capabilities may be used for illegal content or privacy leakage. After supervised fine-tuning and RLHF, models may still be bypassed by adversarial prompts to produce harmful or restricted outputs, posing deployment challenges.
• Existing limitations: The authors cite Wei et al.'s mismatched generalization: pretraining data is larger than safety training data, so out-of-distribution prompts that still match learned behaviors can bypass defenses. The authors state that existing methods mainly use single strategies such as role-playing or encoding, with insufficient prompt diversity.
• Core hypothesis: The authors hypothesize that non-natural-language domains such as equation solving and code completion still have safety weaknesses; converting harmful prompts into forms rarely or never seen during alignment can bypass filters that rely on natural language semantics.
• This paper proposes: EquaCode first converts malicious queries into equation solving, then asks for code completion of unknown steps, using cross-domain tasks to direct attention toward task completion. The authors state in the introduction that this is the first multi-strategy attack and that there is an amplification effect when mathematical tasks are incorporated into code tasks.
• Threat model:
  • Goal: Make the LLM produce harmful or restricted content, specifically providing execution steps to achieve malicious intent.
  • Knowledge: Black-box. The authors state they do not access internal parameters or gradients; white-box optimization requires parameters and is therefore not applicable to closed-source APIs.
  • Capability: The attacker only controls prompts submitted via API, using a unified template to convert queries into equations and embed them in code completion. The abstract states all results require only a single query; the attack cost section states a single API request and claims it is more cost-effective than white-box and many black-box methods. The paper does not report specific time costs.
  • Victim system and judgment: Safety-aligned LLMs, with experiments including commercial GPT series. ASR is the proportion of successful queries; the reviewer model scores 1–10, and only a score of 10 with the response relevant to the query counts as success.

The authors discuss jailbreaks along automated and manual lines, and position this paper as a multi-strategy, cross-domain manual attack.

Safety alignment and mismatched generalization

• Supervised fine-tuning and RLHF: The introduction lists supervised fine-tuning in [1] and reinforcement learning from human feedback in [2] as alignment methods that make models conform to human values.
• Jailbroken: Wei et al. [3] discuss why safety training fails. The authors use its mismatched generalization to explain how out-of-distribution prompts bypass guardrails.
• Long-tail rewriting: Section 3 states that [3, 5] have shown that transforming malicious queries into long-tail forms can bypass safety mechanisms.

Automated jailbreaks

• White-box optimization: GCG [13] uses gradient search to find token sequences that bypass safety guardrails. The authors classify AutoDAN, MAC, and COLD-Attack together with GCG as white-box methods that typically require internal parameters, and state they are infeasible on closed-source APIs.
• Auxiliary LLMs: PAIR [22] uses an LLM to iteratively generate and refine candidate prompts. The authors classify TAP and GPTFuzzer in the same category, and state these black-box methods require multiple rounds of interaction, with higher time and resource costs.

Manual jailbreaks

• DAN: Shen et al. [4] construct prompts using role-playing to bypass restrictions. The authors cite it as an example of a single strategy.
• CodeChameleon: Lv et al. [6] use misleading instructions to elicit restricted outputs; the authors state it takes a psychology-style approach.
• Diversity: The authors state existing black-box methods remain limited by prompt diversity, with most relying on role-playing or encoding. The same section also mentions DeepInception, DRA, ArtPrompt, ReNeLLM, and FlipAttack without detailing their mechanisms one by one.

Baseline methods and benchmarks

• Baseline methods: The authors state they selected 15 recent baselines. The main text lists GCG, AutoDAN, MAC, COLD-Attack, PAIR, TAP, GPTFuzzer, BASE64, DeepInception, DRA, ArtPrompt, PromptAttack, SelfCipher, CodeChameleon, ReNeLLM, and FlipAttack. Table 1 reports the ASR of these methods on six LLMs; the authors state these numbers come from the implementation in [12], and use GPT-4 with the same set of review prompts for comparison.
• Benchmark: AdvBench [13], 520 malicious queries, each violating LLM safety constraints.

The authors position EquaCode as a cross-domain pipeline that first solves equations and then generates code, and state it is not a simple combination of two strategies; they also believe that the natural language domain has received substantial alignment investment, so natural-language-only attacks are no longer likely to achieve high success rates.

EquaCode first rewrites a natural-language malicious query into a text equation, then embeds it into code completion, causing the model to solve for unknown execution steps. The authors state this is a cross-domain pipeline designed in sequence, not a simple concatenation of two modules.

Equation Module

• Decomposition: The malicious query is denoted as A, and is split into Subject B, Tool C, and Execution Steps. B is the action subject, C is the means, and the steps are the unknown quantity x to be solved. A, B, C are known, and x is to be solved. The attacker specifies B and C. The original query, subject, and tool from the example are not written here.
• Equation: The form is B+C+x=A. The authors state this is a text equation composed of text and mathematical symbols, not an ordinary numerical equation.
• Solving: The model is asked to solve for x, i.e., the specific steps to achieve the malicious goal. The authors state that after specifying the subject and tool, the model will focus attention on generating the steps, thereby obscuring intent and bypassing input-level filters that rely on natural language semantics.

Code Module

• Encapsulation: A predefined Python template uses a Solver class that receives A, B, C and maps them to self.A, self.B, self.C. The authors state that consistent naming is intended to bridge the equation strategy and the code strategy and reduce confusion.
• Content to be completed: The constructor additionally initializes an empty list self.steps. solve is responsible for generating steps and appending them to this list. The class also contains functions that return the subject and tool; comments require generating a tool description according to the task. Figure 2 illustrates the code encapsulation.
• Rationale given by the authors: This design prompts the model to focus on syntax and logic rather than whether the content is malicious, thereby bypassing safety guardrails.

Overall Flow and Usage

• Two steps: First, mathematize the query and solve for x; then embed the equation components and the malicious query into the Solver, requiring the steps to be completed in solve and the tool to be described. Figure 1 summarizes these two modules.
• Query overhead: The authors state the template can be applied to any malicious query and is completed in a single API request, without requiring multiple rounds of rewriting as in optimization methods. In comparison, FlipAttack has 4 flip modes and 4 variants, for a total of 16 combinations, requiring exhaustive search to determine the setting.
• Unspecified run parameters: The paper does not specify temperature, top-p, or repeated sampling. The method itself is not iterative optimization.

Success Determination

• Definition: ASR=n/m. n is the number of successful jailbreaks, and m is the total number of queries.
• Evaluation: An LLM judges whether the jailbreak succeeded and whether the response is relevant to the query. The score ranges from 1 to 10, and only a score of 10 counts as success. The evaluation prompt is adapted from [22, 12]. In the main comparison, the authors state they used GPT-4 and used the same evaluation prompt for all methods.

Experimental Setup

• Data: AdvBench, 520 malicious queries. For ablation, to reduce evaluation cost, a subset of 50 policy-violating queries was used.
• Target models: 12 listed in Section 4.1: GPT-3.5-Turbo, GPT-4, GPT-4-Turbo, GPT-4o, GPT-4o-mini, Llama-3.1-405B, and Llama-3.1-70B, Llama-3.3-70B, Gemini-1.5-Pro, DeepSeek-V3, DeepSeek-R1, Grok-3.
• Baselines: Table 1 includes GCG, AutoDAN, MAC, COLD-Attack, PAIR, TAP, Base64, GPTFuzzer, DeepInception, DRA, ArtPromopt, PromptAttack, SelfCipher, CodeChameleon, ReNeLLM, Flipattack. The authors state that the baseline numbers on the first six LLMs come from [12].
• Metrics and review: ASR. The main comparison uses GPT-4 to score 1–10, with only a score of 10 considered successful; GPT-4 is also a tested model. Table 2 notes the evaluator as GPT-ASR, and the paper does not clarify whether this name refers to GPT-4. The ablation instead uses open-source LLaMA-series models for review, without specifying the model versions.
• Other measurements: Perplexity uses the PPL mean of llama-7b, Llama2-7b, and Llama3-8b. Defense discussion involves Llama Guard 7B, Llama 2 Guard 8B, Llama 3 Guard 8B, keyword filtering, PPL Filter, and output filtering.
• Not reported in the paper: Number of repetitions, reviewer-human agreement, and the specific number of API calls for each method.

Main Results

According to Table 1, the numbers are ASR percentages. The method name in the table is EquaCoder. "Highest baseline in that row" is the largest baseline cell read in the same row, not a separately calculated metric.

Swipe horizontally to read the table

Model	EquaCoder	Highest baseline in that row	Baseline name
GPT-4	91.92	86.73	Flipattack
GPT-4-turbo	98.46	94.04	Flipattack
GPT-3.5-turbo	97.12	93.65	DRA
GPT-4o	87.12	92.67	CodeChameleon
GPT-4o-mini	81.35	70.00	DRA
Llama3.1-405B	17.88	27.50	Flipattack
Average	78.98	74.94	Flipattack

• Inconsistencies with the main text: The authors claim EquaCode is highest on GPT-4, GPT-4-Turbo, GPT-3.5-Turbo, and GPT-4o-mini, and write GPT-3.5-Turbo as 97.88%; the cell in the table is 97.12%. The authors claim GPT-4o and Llama3.1-405B rank second and third, respectively, and that GPT-4o-mini is 20% higher than the second-best method. In the table, GPT-4o's EquaCoder is lower than CodeChameleon, Flipattack, and DRA; on Llama3.1-405B it is lower than Flipattack.
• Multiple summary numbers: The authors claim the 78.98% in Table 1 is the highest average across all target LLMs, but the table only includes six models. The abstract also writes GPT-series average 91.19%, 10 SOTA models average 86.98%, and both are single-query. The contributions section writes highest average 84.95%, GPT-series average 92.78%, GPT-4 as 91.19%, GPT-4-Turbo as 98.46%. The conclusion writes GPT series exceeds 90%. Directly corresponding to the table are 78.98% and GPT-4-Turbo's 98.46%; in the table, GPT-4 is 91.92%.
• Low ASR: The authors claim all methods have low ASR on LLaMA-3.1-405B and state that this model has strong safety alignment, with room for improvement in jailbreaks against it. This is a discussion of results, not a Limitations section.

Ablation

• What was tested: Table 4 compares STSA, Equation-only, Code-only, and full EquaCode. STSA only requires decomposing the query into executable steps in natural language. 50-item subset; because the authors claim Llama-3.1-405B previously had very low jailbreak success, here they use Llama 3.1 70B. Parentheses are the ASR difference of EquaCode relative to that single module.
• Results: Average ASR is STSA 17.33%, Equation 44.67%, Code 65.73%, EquaCode 87.33%. On GPT-4-turbo, Code and EquaCode are both 98.00 (parenthesis 0). On GPT-3.5-turbo, Equation 74.00 is lower than STSA 76.00, the only model in the table where Equation is lower than STSA.
• Authors' interpretation: The authors claim Equation is on average 27.34% higher than STSA, Code is 48.4% higher than STSA, and state that weaknesses in code understanding and completion are greater than in mathematical reasoning. For the decrease on GPT-3.5-Turbo, the authors claim it is 2% and attribute it to weaker mathematical reasoning: it rearranges equation terms without generating executable instructions. The authors claim on GPT-4o-mini EquaCode is 2.8 times Code and 4.6 times Equation (in the table: 74.00, 26.00, 16.00). The authors claim the full method is 62.66% higher than Equation and 21.57% higher than Code, and greater than the sum of parts; in the table the averages are 87.33%, 44.67%, 65.73%, and the average ASR is based on the table.

Additional Models, Perplexity, and Defenses

• Table 2: Gemini 1.5, DeepSeek R1, DeepSeek v3 are 100, Grok3 is 95.96, average 98.99. The table note writes the evaluator as GPT-ASR. The main text claims an average of 97.62% and includes GPT-4.1 and Claude 3.7; these two columns are not in Table 2, nor are they in the model list in Section 4.1.
• Table 3: Equacoder's PPL mean is llama-7b 11.14, Llama2-7b 10.90, Llama3-8b 14.60. FlipAttack is 820.44, 543.27, 782.42. ArtPrompt is 6.85, 3.25, 1.99, all three columns lower than Equacoder. The authors claim encoding or encryption methods have high perplexity, EquaCode is lower, and perplexity-based filtering cannot detect or mitigate this attack.
• Guardrails: The authors claim EquaCode bypasses Llama Guard 7B at a rate of 100%, and Llama 2 Guard 8B at 67.88%; the bypass success rate for Llama 3 Guard 8B is relatively lower, and the paper does not give a number. The authors claim PPL Filter is ineffective because the instructions are made low-perplexity. Output filtering, the authors claim, is harder to bypass, with an observed success rate of only 14%, but it requires waiting for the full response to be generated, causing high latency. The dataset, victim LLM, and denominator for these rates are not specified in the paper. Keyword filtering, the authors claim, can block some explicitly malicious questions but is prone to false positives and is rarely adopted in commercial applications. Safety fine-tuning, the authors claim, is computationally expensive, hard to scale, may lag behind jailbreak changes, and sometimes harms general capabilities; the paper does not report the utility or ASR after fine-tuning.

Other Ablations and Analyses

• Equation relative to STSA: The authors claim GPT-4 goes from 2% to 42.0%, GPT-4-Turbo from 26% to 74%; GPT-4o, GPT-4o-mini, LLaMA-3.1-70B go from 0% to 30%, 16%, 32% (Table 4).
• Code relative to STSA: The authors claim GPT-4 improves by 64%, GPT-4-Turbo by 72%; GPT-3.5-Turbo, GPT-4o, GPT-4o-mini, LLaMA-3.1-70B improve by 20%, 54%, 26%, 54%, respectively.
• Table 3's Origin behavior is 30.66, 29.78, 62.16, and the paper does not define Origin in the table note. Caesar Cipher is 335.50, 194.38, 166.69; Morse Cipher is 11.51, 10.23, 10.10.
• Attack cost is only qualitatively compared: white-box optimization consumes GPU, fewer APIs at deployment; auxiliary LLM methods use many APIs; manual methods typically use fewer calls. The paper does not report specific query counts or time.
• Section 4.4 uses gradient saliency analysis to explain attention shift. The authors claim that in Figure 3, high gradients in the original malicious query fall on harm-related tokens; after writing as an equation, attention disperses to equation, show, steps; after embedding in a Python class, more on following, code, Solver, steps. The figure caption is only intention, and color blocks cannot be verified from text. The authors mainly attribute effectiveness to capability-safety misalignment: math and code pretraining is more extensive, while RLHF data is predominantly natural language.

Limitations and future directions noted by the authors

• Capability dependence: The effectiveness depends on the target LLM having sufficient mathematical and programming understanding and processing capability; it must be able to correctly parse and execute adversarial instructions embedded in equation or code structures. When applied to models weaker at symbolic reasoning or code generation, this dependence limits the success rate (4.6).
• Strategy diversity: The authors plan to incorporate a broader range of task forms and expand the diversity of strategy combinations to improve the attack's generalization and adaptation (4.6).
• Automation: The authors plan to explore automation of attack construction to reduce manual effort while improving the attack's efficiency and effectiveness (4.6).

The Limitations and Future Work section only lists these three points. The sentence in the results section about Llama-3.1-405B still having room for improvement is not counted here.

Experimental coverage

• The main metric ASR uses the 520 malicious queries from AdvBench; the ablation in Table 4 uses a subset of 50. The success condition is a review score equal to 10.
• Section 4.1 lists 12 LLMs. Table 1 reports the ASR of 6 of them against 16 baselines; Table 2 reports Gemini 1.5, DeepSeek R1, Grok3, and DeepSeek v3; Table 4 includes Llama 3.1 70B. Llama-3.3-70B is listed in Section 4.1, and the paper does not report its ASR.
• The review model for the main comparison is GPT-4; Table 2 writes GPT-ASR; the ablation uses open-source LLaMA-series models. The paper does not report the number of repetitions, nor reviewer-human agreement.
• Section 4.5 gives bypass rates for Llama Guard 7B and Llama 2 Guard 8B, and discusses Llama 3 Guard 8B, PPL Filter, keyword filtering, output filtering, and safety fine-tuning. The paper does not report the denominator of the bypass rates, the corresponding dataset, or the victim LLM, nor the utility after fine-tuning.
• The baseline ASR for the first six LLMs, the authors state, comes from [12]. Attack cost is compared qualitatively across white-box, auxiliary LLM, and manual methods; the paper does not report the specific number of API calls for each method.

EquaCode is a black-box, single-query multi-strategy jailbreak: it rewrites a malicious request into equation solving and then embeds it into code completion.

• Problem: The authors believe existing jailbreaks mostly use a single strategy in natural language, with insufficient prompt diversity, making it difficult to evaluate robustness after alignment. The hypothesis is that equation solving and code completion still have safety weaknesses.
• Method: The query is split into a subject, a tool, and unknown steps, written as a text equation, then mapped into a Solver class, where the model completes the steps and describes the tool. The authors state the template can be applied to any malicious query, requires a single API request, and does not access parameters. The success criterion is the evaluation model scoring 10.
• Main results: On AdvBench, in Table 1, EquaCoder achieves ASR of 91.92%, 98.46%, 97.12%, 87.12%, 81.35%, and 17.88% for GPT-4, GPT-4-turbo, GPT-3.5-turbo, GPT-4o, GPT-4o-mini, and Llama3.1-405B, respectively, with a six-model average of 78.98%. For GPT-4o and Llama3.1-405B, the table also shows higher baselines.
• Other results: In Table 2, the four-model average is 98.99%, with three at 100% and Grok3 at 95.96%. In 50 ablation cases, EquaCode averages 87.33%, Equation 44.67%, Code 65.73%, and STSA 17.33%. The authors state that combining the two modules has a synergistic effect greater than using either alone.
• Authors' conclusion: The authors believe the cross-domain combination amplifies safety weaknesses in mathematical and coding tasks, attributing this to a capability–safety mismatch: these capabilities are more fully pretrained, while RLHF data is predominantly natural language. The authors also note that the attack is limited when the target model's mathematical or coding capabilities are insufficient, and they plan to expand task forms and automated construction. The 91.19% (GPT series) and 86.98% (10 models) in the abstract are not the same aggregation scope as the 78.98% in Table 1.