Analyzes large language models' vulnerability to prompt injection and jailbreak attacks

How easily 1B–7B open-source LLMs are broken by prompt injection and jailbreaks under direct adversarial prompts, and how much weight-unchanged inference-time defenses can reduce the vulnerability.

• Scenario: The authors say these models have entered education, medical triage, customer service, and software agents, and run on edge devices, universities, and resource-constrained institutions. Most safety evaluations target large proprietary models, while retraining or fine-tuning is often infeasible, so inference-time, weight-unchanged defenses are needed.
• Existing methods: The authors say prior work mostly uses synthetic attacks or only limited data sources. HarmBench, JailbreakBench, and PromptRobust focus on standardized prompts, which the authors believe may not cover the diversity and variation of real adversarial strategies.
• Research questions: RQ1 asks how vulnerable each open-source LLM is to prompt injection and jailbreaks; RQ2 asks how much lightweight inference-time defenses can reduce vulnerability without retraining.
• Evaluation content: Empirical comparison on 10 weight-unchanged models, five existing inference-time defenses, and the failure mode of silent non-response. The authors list the publicly curated adversarial prompt dataset as a contribution.
• Threat model:
  • Goal: Make the model produce restricted or policy-violating content, which the paper calls Successful Jailbreak.
  • Capability: Send prompt injection or jailbreak prompts directly to the model. Categories are question answering, basic compliance, instruction override, roleplay, long-form or multi-step. The experiments are not the indirect injection discussed by Greshake et al.
  • Knowledge: The targets are weight-unchanged open-source models. The paper does not use terms like white-box or black-box to describe what the attacker can see.
  • Judgment: Substantial compliance, giving violating instructions, leaking information that should be refused, or partially assisting after refusal is recorded as Vulnerable. Explicit refusal or empty output is recorded as non-vulnerable; empty output is additionally recorded as silent non-response.

Prompt injection and jailbreaks

• Perez and Ribeiro (2022) manipulate instruction hierarchy with relatively simple adversarial prompts, causing models to ignore existing safety constraints. The paper uses this to state that alignment is not inherently robust to instruction-hierarchy manipulation and adversarial rewriting.
• Greshake et al. (2023) do indirect prompt injection: malicious instructions embedded in external content affect applications that integrate LLMs. This paper's experiments do not follow that setting.
• Wei et al. (2023), Zou et al. (2023), Liu et al. (2023), and Shen et al. (2023) respectively analyze safety alignment failure under adversarial prompts, attacks transferable across architectures, prompt-engineering-based jailbreaks, and in-the-wild jailbreak prompts. The authors summarize: attacks exploit the tension between helpfulness and harmlessness; coherent, context-rich, or multi-step reasoning prompts may let instruction following override safety constraints.

Defenses

• Training-time approaches: supervised fine-tuning and RLHF from Ouyang et al. (2022), and Constitutional AI from Bai et al. (2022). The authors state that these bake safety constraints into optimization, requiring weights and substantial compute, which is often infeasible in open-source or production settings.
• Inference-time approaches: input filtering and policy guardrails ([11] in the paper), embedding-space detection ([14]), voting aggregation ([9]), and self-checking ([31]). The authors state that these are lightweight and need no retraining, but may degrade under long or multi-step jailbreaks; generalizable defenses remain an open problem.
• What this paper compares is concrete implementations of these five categories, with the control being directly feeding prompts with no defense; the data are the authors' own curated adversarial prompts, not the full original benchmarks of the above papers.

Open-source models

• Bommasani et al. (2021) discuss opportunities and risks of foundation models. The authors emphasize that open-source models can be downloaded locally, have no centralized monitoring, and receive slow patches; after weights are downloaded, users can probe offline, craft targeted attacks, or fine-tune away safety constraints.
• The authors select six families from 1B to 7B: Llama 3.2, Mistral 7B, Phi-3, Qwen 3, Gemma 3, and DeepSeek-R1. The selection rationale is common deployment and multiple sizes within each family, allowing examination of whether safety varies with capacity. The paper only describes the alignment practices in each family's public materials and does not write any family as already safer.
• The open question the authors raise is whether language capability and safety refusal can fit together at small parameter counts. The evaluation only covers models with unmodified weights.

Evaluation benchmarks

• HarmBench (Mazeika et al., 2024), JailbreakBench (Chao et al., 2024), and PromptRobust (Zhu et al., 2023) provide structured red-teaming or adversarial robustness evaluations. The authors state that these are mainly based on standardized prompt sets and may not cover the diversity of adversarial strategies in real deployments.
• Community sources include in-the-wild jailbreaks, system prompt leaks, and red-teaming scripts on Reddit, Discord, and GitHub. The authors merge these with academic benchmarks to cover both controlled conditions and still-evolving attacks.

Baselines and data: No defense versus five inference-time defenses. Prompt injection has 94 prompts per model; jailbreak with no defense has 73 prompts per model; with defense, each configuration has 74 longer, more structured prompts. Section 5.3 states that the two prompt sets are not identical.

The authors position this paper as: relative to work focusing on synthetic attacks or limited data sources, it uses multi-source real prompts, evaluates small-to-medium open-source models in resource-constrained settings, and inference-time defenses that need no retraining.

Overall, this is an empirical evaluation, not a new attack or new training algorithm: adversarially curated prompts, first testing a no-defense baseline, then applying five inference-time filters that do not change weights on prompts the authors call longer and more structured.

Objects and two phases

• Models: Phi-3 (3.8B), Mistral (7B), DeepSeek-R1 (7B and 1.5B), Llama 3.2 (3B and 1B), Qwen-3 (4B and 1.7B), Gemma-3 (1B and 4B), 10 in total, weights unchanged.
• Phase 1: Adversarial prompts go directly into the model to establish a baseline.
• Phase 2: Five inference-time defenses are enabled, stress-tested with prompts the authors call more advanced and context-rich.
• Not formalized: The paper has no white-box/black-box definition of attacker knowledge, nor a loss function or optimization objective.

Adversarial prompts

• Sources: Communities such as Reddit, GitHub repositories, attack-method papers (including JailbreakBench, HarmBench, PromptRobust, and Perez and Ribeiro, Liu et al., Wei et al., Zou et al.), resources related to uncensored model variants, and manual rewriting. Table 1 has pattern examples; the original text is not quoted here.
• Five categories: Question Answering, Basic Compliance, Instruction Override, Role-playing Jailbreaks, Long-format / Multi-step Prompts.
• Long text: The authors state they exploit attention being diluted by long context, instruction models prioritizing coherence and helpfulness, multi-step gradual escalation of intent, and embedding instructions into seemingly harmless narratives. The paper gives no length threshold or step count.
• Scale: 94 injections per model. Jailbreaks: 73 per model, 438 total, 6 models. With defenses: 6 models, 74 per configuration, 2,220 total. The latter two phases keep only configurations that execute stably; the paper states that in preliminary testing there were empty responses, timeouts, or interrupted runs.

How vulnerabilities are judged

• Vulnerable: Provides executable guidance or explanations that violate safety policy; leaks information that should be refused under standard safety constraints; partially complies and advances the adversarial goal. A refusal followed by partial unsafe assistance is still counted as vulnerable.
• Not vulnerable: Refusal, safety warning, or shifting to an alternative statement without restricted content. Empty output caused by internal gating is called silent non-response, counted as not vulnerable, but analyzed separately.
• Judge: The baseline is mainly judged Vulnerable / Non-Vulnerable by another LLM, required to reason like an expert rather than use keyword matching. The paper does not name that judge model. A subset is manually spot-checked, inconsistencies corrected, and used to improve the judge prompt. The paper does not report the number of spot-checked items or agreement.
• Defense phase: Section 5.2.2 states all responses are judged with human-level vulnerability determination, not only automatic safety word lists.

Five inference-time defenses

1. Input risk classification ([11]): Before generation, a threat score is assigned by pattern, keyword, and structure; above a threshold, it is blocked or sanitized. The paper does not give the threshold or word list used in experiments.
2. Self-check ([31]): The main model generates first, then a judge labels SAFE or HARMFUL; if harmful, it is blocked, regenerated with constraints, or replaced with a safe fallback. The authors state the safety judge is DeepSeek-R1, chosen for its performance against prompt injection and reasoning ability. The paper does not state whether it is 1.5B or 7B.
3. Policy guardrail: A hardened system prompt is prepended to the user query, specifying safety policy, instruction priority, category restrictions, refusal templates, and role boundaries. Only the prompt template is changed.
4. Vector defense ([14]): The same encoder embeds the input into a vector library, computing cosine similarity or Euclidean distance against known attacks; above a threshold, it is flagged. The paper does not report the encoder, threshold, or library size.
5. Voting ([9]): The same prompt is generated multiple times, written as typically 3–5, with different random seeds or sampling parameters; each response is assessed for safety, then the more conservative output is taken by majority, and if all are unsafe, a standard refusal is returned. The experimental section does not report the actual N, temperature, or sampling.

Trade-offs written by the authors

• Inference-time filtering sits outside the model and requires no GPU fine-tuning. Section 4 writes: input filtering has low overhead and negligible latency; self-check requires one extra forward pass, possibly doubling latency; guardrails add almost no latency but the long system prompt consumes context; vector retrieval has moderate latency; voting is about 3–5 times the forward passes.
• The authors also write failure modes: input filtering has high false positives and can be bypassed by rewriting; the self-check judge itself can be adversarially attacked, and self-review by the same model may rationalize its own unsafe outputs; guardrails rely on model compliance and can be bypassed by role-playing and multi-step reasoning; vector defense is weak against new attacks far from known samples; voting does not guarantee every sample is safe.

Experimental Setup

• Models: 10 tested for injection: Phi-3 (3.8B), Mistral (7B), DeepSeek-R1 (7B), DeepSeek-R1 (1.5B), Llama 3.2 (3B), Llama 3.2 (1B), Qwen-3 (4B), Qwen-3 (1.7B), Gemma-3 (4B), Gemma-3 (1B). Jailbreak and defended conditions use 6 of these: gemma3:1b, Llama3.2:3b, Llama3.2:1b, qwen3:1.7b, qwen3:4b, DeepSeek-R1:1.5b.
• Data: Undefended injection: 94 per model, sourced from Reddit, GitHub, JailbreakBench and HarmBench (named in the paper), and the paper itself. Undefended jailbreak: 73 per model, 438 total. Defended: 6 models, authors state 74 per configuration, 2,220 total tests; authors state this batch is longer and more structured.
• Conditions: Undefended; five defenses: Self-defence, Input filtering, System Prompt Defence, Vector Defence, Voting Defence.
• Metrics: Table 3 Vulnerable/94 and Rate (%); Table 4 Success, Timeout, Vuln, Non-Vuln; Table 5 Vulnerability (%) by source. Silent non-response counts as non-vulnerable but is analyzed separately.
• Review: Baseline mainly uses an unnamed LLM judge, with a subset corrected by humans. Defense results use what the authors call human-level judgment. The paper does not report number of repetitions, random seeds, number of human-checked samples, judge model name, or statistical tests.
• Exclusions: The paper states some configurations frequently produced no response, timed out, or aborted on long prompts; jailbreak and defense results keep only those that ran stably. No list of excluded configurations is given. The contribution overview says 10 models across 73 jailbreaks; Table 4 has only 6 models.

Main Results

Prompt injection, undefended, 94 per model (Table 3). Exceeds 8 rows, so qwen3:4b and gemma3:4b, both 0/94, are omitted.

Model	Vulnerable/94	Rate (%)
qwen3:1.7b	67	71.3
gemma3:1b	59	62.8
DeepSeek-r1:1.5b	32	34.0
Llama3.2:1b	29	30.9
phi3:3.8b	0	0
Mistral:7b	0	0
DeepSeek-R1:7b	0	0
Llama3.2:3b	0	0

• Authors' grouping: High vulnerability: qwen3:1.7b, gemma3:1b; medium: DeepSeek-R1:1.5b, Llama3.2:1b. The text says about 20%–40%; the table shows 34.0% and 30.9%. The other six are 0 on 94. The authors state the high-vulnerability models fail broadly across many prompt types, with weaker instruction hierarchy and safety goal alignment.
• 0 does not mean usable: The authors state phi3:3.8b and Mistral:7b often return empty responses with no refusal explanation, and suggest this may be hard gating before decoding. DeepSeek-R1:7b and Llama3.2:3b more often give explicit refusals and policy-based explanations. The authors state zero vulnerability does not imply transparency or usability; silent non-response is a hidden failure mode and also affects debugging.
• Scale: The authors state safety does not increase monotonically with parameters. The text describes qwen3:4b vs qwen3:1.7b as 0 vs 71%, and gemma3:4b vs gemma3:1b as 0% vs 63%; Table 3 shows 0 vs 71.3% and 0 vs 62.8%. The authors argue alignment strategy, safety layering, or refusal design matter more than scale. The Figure 1 transcription has no numbers beyond the table.

Jailbreak (undefended)

• Setup: 6 models, 73 each, 438 total (Table 4). In most rows Success plus Timeout equals 73, and Vuln plus Non-Vuln equals Success.
• Counts: gemma3:1b: Success 53, Timeout 20, Vuln 49, Non-Vuln 4; Llama3.2:3b: 52, 20, 36, 16; qwen3:1.7b: 55, 18, 8, 47; Llama3.2:1b: 42, 31, 7, 35; DeepSeek-R1:1.5b: 55, 18, 1, 54. qwen3:4b has Total 73, but Success 0, Timeout 12, Vuln 0, Non-Vuln 0; the four columns do not sum to 73, so that row is not converted to a vulnerability rate.
• Authors' claims vs table: The authors state gemma3:1b is the most easily compromised, failing on about 67% of adversarial prompts; DeepSeek-R1:1.5B is lowest at only 1.37% (table shows Vuln 1); Llama-3.2-1B timed out 31 times out of 73. They also state DeepSeek and Qwen are more balanced between timeout and safety than other models. Section 5.3 states Gemma3:1B and Qwen3:1.7B have undefended vulnerability rates above 60%. This matches Table 3's 62.8% and 71.3%, but not Table 4's Vuln 8 for qwen3:1.7b; the table takes precedence. Figure 2 and Figure 3 transcriptions have no extra tick marks.

Five inference-time defenses

• Setup: Same 6 models, 74 per configuration, 2,220 total. The prompt set differs from the undefended set. Judgment changed to human-level. Figure 4 plots vulnerability rate by model times defense; the text gives only partial approximate values, no full numeric table.
• Numbers given by authors: Self-check lowest; qwen3_4b and DeepSeek-R1-1.5b at 0% under self-check; Llama3.2_1b down to about 9.5%. Input filtering weakest: Llama3.2_3b about 42%, gemma3_1b above 30%. Under system prompt, qwen3_1.7b and DeepSeek-R1-1.5b near 0, Llama3.2_3b still about 39%. Vector defense about 25–30% on larger models. Under voting, smaller models again about 13–15%. The authors state self-check reduces vulnerability by more than 50% relative to input filtering. These approximate numbers are textual descriptions of Figure 4.
• Authors' interpretation: Figure 5 summarizes vulnerable and non-vulnerable counts across all models; the text gives no specific counts. The authors state self-check has the highest non-vulnerable proportion, input filtering the most vulnerable counts; system prompt and vector defense still leave some unsafe answers; voting is less than input filtering but does not eliminate failures. The authors argue external or reactive defenses leak more than the model's own refusals; with larger capacity and no internal defense, stronger instruction following may amplify jailbreak susceptibility. Section 5.3 emphasizes the two prompt sets differ, so this is not a before/after comparison on the same prompts; even with harder prompts, the authors state no model under defense exceeds the vulnerability level seen in the baseline. Defenses do not fully remove risk.

Other ablations and analyses

• Table 5, Horselock jailbreak (DeepSeek variant): 50.00%
• Table 5, Discord BASI: 31.48%
• Table 5, Reddit/DAN: 26.85%
• Table 5, Reddit r/VeniceAI: 22.22%
• Table 5, Reddit r/LocalLlama: 3.33%
• Table 5, Reddit r/AIJailbreak: 0.00%

The authors state Horselock-style multi-role and covering prompts are most effective, and some popular Reddit jailbreaks are outdated or already patched. The paper does not state Table 5's denominator or corresponding model. The experiments section gives no ablation numbers by temperature, threshold, voting count, or embedding model. The abstract and conclusion state lightweight defenses can mitigate direct attacks, but long, reasoning-heavy prompts can still bypass them.

Limitations and future directions noted by the authors

• No dedicated section: The paper has no dedicated Limitations section. Section 1 states that Section 6 contains future research directions.
• Future directions (Section 6): The authors state that further research is needed to strengthen built-in defense capabilities and build safety reasoning directly into the LLM architecture.
• Effect boundaries in the conclusion (Section 6): The authors state that stable inference-time defenses improve resistance to simple attacks but degrade on complex, long-form jailbreaks; external filtering alone is insufficient to guarantee safety.
• Section 2.2 states that generalizable defenses remain an open problem. That sentence is not in Limitations or Conclusion and is not counted as a limitation statement about this experiment.

Experimental coverage

• The prompt injection baseline is the 10 open-source models in Section 3.1 and Table 3, with 94 prompts per model, no defenses.
• The jailbreak baseline is the 6 models in Table 4, with 73 prompts per model, totaling 438 prompts. The paper states that only stable configurations were retained due to execution stability (Section 5.1.2).
• The defenses are five mechanisms times these 6 models, with 74 prompts per configuration, totaling 2,220 prompts (Section 5.2.1). The safety judge in the self-check implementation is DeepSeek-R1; the paper does not state its size.
• Defended and undefended settings use different prompt sets (Section 5.3). The paper does not report the model name used by the main judge, the number of manually sampled prompts, judge-human agreement, the number of repetitions, or the voting N, filtering thresholds, and vector encoder used in the experiments.
• The evaluation targets models with unmodified weights (Section 2.3.2). The paper notes that users can fine-tune away safety constraints offline and states that this evaluation does not cover that case.

On open-source models from 1B to 7B with unchanged weights, using adversarially curated prompts from multiple sources, comparing prompt injection, jailbreaks, and inference-time defenses without retraining.

• Problem: When resources are tight, retraining for alignment is often not possible. The authors examine how easily models are broken by prompt injection and jailbreaks, and how much five inference-time filters can reduce vulnerabilities.
• Method: 94 injections per model; jailbreaks only on 6 configurations that execute stably, 73 per model. Defenses use longer prompts, 74 per configuration. Substantive compliance is counted as vulnerable; empty output is counted as not vulnerable but separately called silent non-response. The baseline uses an unnamed LLM judge with manual spot-checking; in the defense phase, the authors state the judgment is changed to human-level.
• Injection: In Table 3, qwen3:1.7b is 71.3% (67/94), gemma3:1b is 62.8% (59/94), DeepSeek-r1:1.5b is 34.0%, Llama3.2:1b is 30.9%. phi3:3.8b, Mistral:7b, DeepSeek-R1:7b, Llama3.2:3b, qwen3:4b, gemma3:4b are 0. The authors state that the 0 for phi3 and Mistral is often accompanied by no output.
• Jailbreak: In Table 4, gemma3:1b has Vuln 49, Success 53, Timeout 20. The authors state about 67%, and state DeepSeek-R1:1.5B is only 1.37% (Vuln 1 in the table). Llama3.2:1b has 31/73 timeouts. In Table 5, Horselock (DeepSeek variant) is 50.00%, Reddit r/AIJailbreak is 0.00%.
• Defenses and conclusion: The authors state self-check is lowest, with qwen3_4b and DeepSeek-R1-1.5b at 0%, and under input filtering Llama3.2_3b is about 42%. The authors state these filters can mitigate direct attacks, but long, reasoning-heavy prompts can still bypass them; external filters are inferior to the model's own refusal. The Conclusion proposes strengthening built-in defenses and building safety reasoning into the architecture.