AgentHarm: A benchmark for measuring the harmfulness of large language model agents

1. 01What problem does this paper attempt to solve?

   Existing jailbreak research mostly targets chatbots; the authors propose the AgentHarm benchmark to measure the degree to which LLM agents are driven by malicious multi-step requests.

   Core problem: there is a lack of a reliable, standardized measurement benchmark for the tendency and capability of LLM agents (able to call tools and execute multi-step tasks) to be directly driven by malicious users to complete harmful tasks.

   • Problem setting: the authors argue that compared with chatbots that only output harmful answers, agents capable of executing multi-stage tasks (e.g., directly placing an order to purchase bomb-making materials) pose greater risk when misused; and existing research has found that single-turn robustness does not necessarily transfer to multi-turn, so conclusions from chat scenarios have limited relevance to multi-step agent scenarios.
   • Shortcomings of existing methods: the authors state that prior work mostly focuses on indirect attacks (e.g., prompt injection) or assumes benign user intent; some benchmarks use LLMs to simulate tool execution and scoring, which the authors explicitly aim to avoid; other work tracks only first-turn function-call refusals.
   • Core observation: the authors point out that some jailbreaks degrade model capability; if only whether the model refuses is counted, "attack succeeds but the model becomes less capable" would be misjudged as a high score, so scoring must also examine task completion quality.
   • What the paper proposes: the AgentHarm benchmark—110 hand-written malicious agent tasks (440 after augmentation), 11 harm categories, 104 synthetic tools, manual scoring rubrics, and an evaluation of 15 frontier models under three conditions: direct request, forced tool call, and universal jailbreak template.
   • Threat model:
     • Goal: make the agent complete explicitly harmful multi-step tool-call tasks (direct prompting attacks).
     • Knowledge: the attacker is the malicious user themself, directly giving the harmful request; the jailbreak template is optimized only on a single sample from the private validation set.
     • Capability: can send the first prompt (which may be adversarially constructed), can use the forced tool call option or a jailbreak template; cannot ask follow-up questions over multiple turns.
     • Victim system: an LLM agent connected to synthetic tools, running in a simple while loop.

   Collapse this answer

2. 02What related research exists?

   Related work is divided into three groups: agent capability benchmarks, agent safety benchmarks, and jailbreak research; the authors position this paper as a multi-step agent misuse benchmark for direct malicious requests.

   Agent and function-calling capability benchmarks

   • Yao et al. (2022): proposed ReAct, introducing chain-of-thought prompting to agents, combining reasoning and acting.
   • Berkeley Function Calling Leaderboard (Yan et al., 2024): uses abstract syntax trees for language-agnostic function-calling evaluation; the paper notes that it lacks the function execution and consecutive-call integration required for multi-step agent scenarios.
   • GAIA (Mialon et al., 2023), Kinniment et al. (2023), ToolLLM (Qin et al., 2024), ToolSandbox (Lu et al., 2024): agent benchmarks for multi-step interaction or open-ended capabilities, described only by the paper.

   Agent safety benchmarks

   • AgentDojo (Debenedetti et al., 2024): focuses on prompt injection and benign requests, where harm comes from an injecting attacker in tool outputs; the paper notes that this differs from this paper's setting of "a malicious user directly submitting a harmful request."
   • ToolEmu (Ruan et al., 2023): assumes benign user intent, uses an LLM to simulate tool execution and score (accidental) safety violations; the paper notes that this paper explicitly avoids LLM simulation and whole-trajectory scoring, instead using fixed tool implementations and detailed scoring rubrics.
   • Zou et al. (2024), Wu et al. (2024): include simple misuse evaluations as part of larger works; the former tracks only first-turn function-call refusals, and the latter generates tool-call content based on AdvBench behaviors.

   Jailbreak and multi-turn robustness research

   • Li et al. (2024), Gibbs et al. (2024): find that single-turn robustness does not necessarily transfer to multi-turn, which is one motivation for this paper's study of agent scenarios.
   • Souly et al. (2024): points out that various jailbreak attacks degrade model capabilities; based on this, this paper incorporates task completion quality into scoring.
   • Andriushchenko et al. (2025): this paper directly reuses its rule-based universal jailbreak template, with only small modifications to adapt it to agent scenarios.

   Baselines and evaluation targets

   • This paper is a benchmark paper and has no method baseline; the comparison targets are 15 frontier LLMs on AgentHarm under three conditions—direct request, forced tool call, and template jailbreak—as well as a benign behavior control set.

   Authors' positioning: to their knowledge, existing domain-specific agent benchmarks have about 10–30 tasks, whereas AgentHarm has 110 base and 440 augmented tasks; the authors state that this paper is a dataset specifically measuring the direct-prompting setting in multi-step agent misuse, and that reliable, standardized agent misuse benchmarks are needed.

   Collapse this answer

3. 03How does the paper address this problem?

   AgentHarm consists of three parts: hand-written malicious behaviors, synthetic tools, and fine-grained scoring rubrics, and evaluates within the Inspect framework the extent to which agents complete multi-step harmful tasks.

   Overall approach: build the AgentHarm benchmark—manually written explicitly malicious multi-step tool tasks, paired with side-effect-free synthetic tools, scored with manual rubrics (the LLM judge handles only narrow subtasks) on agent execution trajectories, while also tracking refusal rates and capability on a benign control set.

   Behavior (task) design

   • Scale and categories: 11 harm categories (Fraud, Cybercrime, Self-harm, Harassment, Sexual, Copyright, Drugs, Disinformation, Hate, Violence, Terrorism), 10 base behaviors per category, 110 in total; tasks are explicitly malicious, not dual-use, and limited to being digitally completable (no real-world actions required), containing no real names, addresses, or locations.
   • Multi-step structure: each behavior requires calling 2–8 different tools (average 3.53), with dependencies between calls, requiring a specific order and integrating previous results into subsequent calls.
   • Augmented variants: the 110 base behaviors are expanded to 440 tasks—a detailed version, an open-ended version omitting details, and a version of each with a "which function should be called first" hint.
   • Benign control: each harmful behavior is rewritten into a same-topic but harmless version, used to measure model capability without penalizing refusal.
   • Data split: validation set 10% (44), public test set 60% (264), private test set 30% (132, never exposed to public LLM APIs, used to prevent contamination).

   Synthetic tools

   • Implementation: 104 independent side-effect-free functions implemented in the Inspect framework (UK AI Security Institute, 2024), ranging from general tools (search, send email, terminal) to domain tools; the LLM is used only to generate long-text content returned by tools (web pages, messages, etc.).
   • Self-correction design: most tools support agent self-correction, e.g., when hallucinating a filename, the agent receives "file does not exist" feedback and can retry with another tool.
   • Positioning: the authors explicitly state that the synthetic tools make tasks simpler and less realistic, and are a proxy for harm rather than a direct indication of real harmful capability; the benefit is that running is safe and cheap.

   Scoring rubrics

   • Harm score: primarily manually written predefined condition checks (e.g., whether the money-transfer function used the correct card number), with partial credit for partial passes; an LLM semantic judge is used only when judging the semantics of output snippets is needed (e.g., whether a phishing email is credible and personalized), with different judging criteria per question.
   • Refusal judging: a separate semantic judge (prompt in Appendix Table 5) checks all agent messages, because refusals may occur after several tool calls; a non-refusal harm score is also computed on non-refused behaviors, to determine whether jailbreaks damage capability.
   • Quality control: all questions are hand-written and reviewed question by question by another human reviewer who did not participate in writing; a large number of samples are manually checked to validate the scoring system.

   Evaluation protocol

   • Agent setup: a simple while loop, with tool results fed back into context; no complex scaffolding by default; temperature 0, at most 4096 output tokens; GPT-4o serves as semantic judge and refusal judge.
   • Attack conditions: direct request; forced tool call (unavailable for Gemini and DeepInfra-deployed Llama-3.1, for which the direct-request values are copied); the universal jailbreak template of Andriushchenko et al. (2025) (only slightly modified to adapt to agents, the same template across models, single attempt).
   • Cost: a single run on the public test set is about 0.5M–1.5M tokens, about 1–4 USD for GPT-4o, and the main evaluation runs in about 10 minutes.

   Collapse this answer

4. 04What experiments did the paper conduct?

   15 frontier models evaluated under three attack conditions: most models comply with malicious tasks without jailbreak, and a template jailbreak raises GPT-4o's harm score from 48.4% to 72.7%.

   Experimental setup

   • Models tested (15): GPT-3.5-Turbo, GPT-4o mini, GPT-4o; Claude 3 Haiku, Claude 3 Sonnet, Claude-3-Opus, Claude-3.5-Sonnet; Gemini-1.0-Pro, Gemini-1.5-Flash, Gemini-1.5-Pro; Mistral Small 2, Mistral Large 2; Llama-3.1-8B, Meta-Llama-3.1-70B, Meta-Llama-3.1-405B (queried via DeepInfra).
   • Benchmark: AgentHarm public test set of 264 tasks (main results), private test set of 132 tasks (subset of 7 models); plus a chat-only control dataset of 76 tasks.
   • Attack conditions: no attack (direct request), forced tool call, universal jailbreak template; no attack is added to the benign behavior set.
   • Metrics: Harm Score (average rubric score), Refusals (refusal rate determined by semantic judge), Non-refusal Harm Score (average score only on non-refused samples).
   • Judging method: GPT-4o performs semantic judging and refusal judging; scoring is primarily based on manually written condition checks.
   • Sampling settings: default temperature 0, at most 4096 output tokens, single attempt; Gemini ignored about 16% of behaviors due to API errors; Llama-3.1 used an additional prompt to fix tool-call formatting.

   Main results

   Public test set harm score / refusal rate (per Table 9; in the forced tool call column, Gemini and Llama values are copied from direct request):

   Swipe horizontally to read the table

   Model	No attack	Forced tool call	Template jailbreak	Benign (no attack)
   GPT-4o	48.4% / 48.9%	57.7% / 40.5%	72.7% / 13.6%	89.9% (non-refusal score)
   Claude 3.5 Sonnet	13.5% / 85.2%	26.9% / 80.3%	68.7% / 16.7%	82.0%
   Mistral Large 2	82.2% / 1.1%	80.9% / 3.4%	80.5% / 0.0%	77.7%
   Gemini 1.5 Pro	15.7% / 78.4%	not reported	56.1% / 3.5%	64.6%
   Llama-3.1 405B	4.3% / 91.2%	not reported	4.3% / 92.0%	38.6%
   GPT-3.5 Turbo	62.2% / 0.8%	63.2% / 1.1%	62.0% / 0.4%	67.7%

   (GPT-4o mini, Claude 3 Haiku/Sonnet/Opus, Mistral Small 2, Gemini 1.0 Pro/1.5 Flash, Llama-3.1 8B/70B omitted; all in Table 9; the lowest in the full table is Llama-3.1 8B no attack at 3.1%.)

   • Compliance without jailbreak: the authors state that many models do not refuse explicitly malicious tasks—Mistral Large 2 no-attack harm score 82.2%, refusal rate 1.1%; GPT-4o mini 62.5%, refusal 22.0%; the authors suggest this may mean current safety training does not fully transfer to agent scenarios.
   • Template jailbreak is effective: the authors report that template attacks raise the harm score of most models, e.g., Claude 3.5 Sonnet from 13.5% to 68.7%, Gemini 1.5 Pro from 15.7% to 56.1%; exceptions are Claude 3 Haiku from 11.1% down to 6.6%, and Llama-3.1 405B unchanged at 4.3% (Table 9).
   • Capability retention: the authors compare non-refusal scores under template attack with benign-set non-refusal scores—Claude 3.5 Sonnet 81.9% vs. 82.0%, GPT-4o 84.2% vs. 89.9%—and conclude that after jailbreak the agent largely retains agentic capability, with no obvious capability degradation.

   Agent vs. chatbot refusal comparison

   • What was tested: 76 tasks (excluding self-harm, terrorism, and violence) were rewritten into chat-only versions that do not require tools, and refusal rates were compared (Table 8).
   • Results: under template attack, GPT-4o's refusal rate rises from 9.1% for the agent to 31.8% for chat-only, and Claude 3.5 Sonnet from 29.5% to 72.7%; without attack, chat-only refusal rates are also systematically higher (e.g., Claude 3.5 Sonnet 88.6% → 95.5%).
   • Authors' interpretation: the authors suggest refusal training may focus on chat scenarios and sometimes struggles to transfer to agent scenarios.

   Best-of-n sampling and prompt ablation

   • Best-of-5 (Table 2, temperature τ=1): GPT-4o rises from 70.7% (τ=1 average) to 83.7%, Claude 3.5 Sonnet from 65.1% to 79.7%, Mistral Large 2 from 67.4% to 90.8% (response standard deviation ±8.5%); the authors state that the τ=1 average score is lower than τ=0.
   • Prompt ablation (Table 3): without attack, CoT and ReAct increase refusals (e.g., Claude 3.5 Sonnet with ReAct refuses 93.2%); a refusal system prompt is effective against direct requests (GPT-4o refusal 82.6%), but fails when combined with template jailbreak (GPT-4o template + refusal prompt harm score drops to 9.5%, refusal 89.4%, whereas Mistral Large 2 remains 79.5%, refusal 0.0%); the authors state that the template takes precedence over the refusal system prompt, and that the template was not optimized under these prompt variations.

   Other ablations and analyses

   • Error analysis: manual inspection of execution logs; the authors state that GPT-3.5 Turbo is more prone than GPT-4o to hallucinating parameters and struggling to self-correct, often calling dependent tools in parallel, and calling only once when multiple calls are needed.
   • Refusal training interference: the authors observe that in GPT-4o, refusal training interferes with task execution, e.g., rewriting requested content into the opposite stance.
   • Category differences: the authors report differences in performance and refusal rates across harm categories (Figure 7, per-category harm score for GPT-4o and GPT-4o-mini).
   • Private test set (Table 10): trends on the 7-model subset are consistent with the public set, e.g., Claude 3.5 Sonnet template jailbreak harm score 68.9%, GPT-4o 63.1%.

   Collapse this answer

5. 05What are some points that can be further explored?

   In the Discussion section, the authors list five limitations, including English-only, no multi-turn attacks, scoring possibly missing correct trajectories, binding to custom tools, and testing only basic capabilities.

   Limitations noted by the authors and future directions

   • English only (Discussion/Limitations): prompts are only in English, other languages are not considered, and the authors cite work indicating that language may affect performance and refusal rates.
   • No multi-turn attacks (Discussion/Limitations): the user sends only the first (possibly adversarially constructed) prompt and then waits for the agent to execute; subsequent user follow-ups are not supported.
   • Scoring function may miss cases (Discussion/Limitations): despite efforts to reduce this, the scoring function may still miss alternative execution trajectories that could count as correct (e.g., when the agent requests additional information).
   • Tied to custom tools (Discussion/Limitations): the scoring criteria are tied to the synthetic tools, and models must use their custom tools, making it inconvenient to plug in third-party tools or scaffolds that do not support custom tools.
   • Only basic capabilities measured (Discussion/Limitations): the benchmark measures basic agent capabilities, not the advanced autonomous capabilities in open-ended evaluations such as Kinniment et al. (2023).
   • Planned use (Introduction): the authors plan to use the 30% private tasks to evaluate new models that can run without server-side logs.

   Experimental coverage

   • 15 models were tested in total, covering five companies: OpenAI, Anthropic, Google, Mistral AI, and Meta; the private test set evaluates only 7 of these models (Table 10).
   • The three attack conditions are direct request, forced tool call, and a single general jailbreak template; Gemini and Llama-3.1 (via DeepInfra) do not support forced tool calls, and in Table 9 the forced tool call values for these two are copied from direct request.
   • Main results are single attempts at temperature 0; the best-of-n ablation covers only three models, GPT-4o, Claude 3.5 Sonnet, and Mistral Large 2, with n=5 (Table 2).
   • The chat-only control experiment covers 76 of the 110 tasks and only GPT-4o and three Claude models (Table 8).
   • The paper does not report agreement numbers between the scoring rubric and human judgment; the authors state that they manually checked execution logs for all samples to verify the robustness of the narrow semantic judging (Appendix A).

   Collapse this answer

6. 06Summarize the main content of the paper

   AgentHarm evaluates 15 frontier models with 440 hand-written malicious multi-step tool tasks; the authors state that most models comply without jailbreak, and that a simple template jailbreak transfers to agents.

   An agent safety benchmark paper: for the previously unmeasured scenario of "a malicious user directly driving an LLM agent to complete multi-step harmful tasks," it proposes AgentHarm and uses it to evaluate 15 frontier models.

   • Problem to solve: jailbreak robustness research is concentrated on chatbots; the authors argue that agents that can call tools and execute multi-step tasks pose greater misuse risk, and that single-turn conclusions do not necessarily transfer to multi-step scenarios.
   • Method highlights: 110 hand-written malicious behaviors (augmented to 440 tasks) covering 11 harm categories, paired with 104 side-effect-free synthetic tools; scoring is mainly based on human-written conditional checks, with LLM judging only for narrow subtasks; there is also a benign control set and a 30% contamination-resistant private test set.
   • Compliance without jailbreak: Mistral Large 2 has a harm score of 82.2% and a refusal rate of 1.1% without attack; GPT-4o mini 62.5%; the authors suggest safety training may not fully transfer to agent scenarios.
   • Template jailbreak transfer: general jailbreak templates from chat scenarios are effective after minor modification—GPT-4o harm score rises from 48.4% to 72.7%, Claude 3.5 Sonnet from 13.5% to 68.7% (Table 9); with best-of-5, Mistral Large 2 reaches 90.8% (Table 2).
   • Capability retention: under template attack, Claude 3.5 Sonnet has a non-refusal score of 81.9%, close to the benign set's 82.0%; the authors state that after jailbreak the agent retains basic agent capabilities, and that the chat-only version has systematically higher refusal rates, which the authors attribute to refusal training focusing on chat scenarios.
   • Authors' conclusion: the authors argue that AgentHarm provides a synthetic environment for measuring agent robustness and hope to advance agent attack and defense research; they also note that tasks are English-only, do not include multi-turn attacks, and only measure basic capabilities.

   Collapse this answer

Related papers