The paper proposes a multi-turn online harassment agent benchmark; jailbreak fine-tuning raises the attack success rate above 95%

How to measure the extent to which aligned LLM agents are jailbroken in multi-turn online harassment.

• Scenario: The paper cites Pew Research: roughly 41% of U.S. adults report experiencing online harassment, 25% say it is severe, and it disproportionately targets women, people of color, people with disabilities, and other groups. The authors state that as LLMs enter virtual assistants, customer service, and autonomous agents, long-horizon interaction, tool use, and memory amplify harassment; once guardrails fail, agents can reproduce harmful multi-turn behavior at a speed, volume, and persistence exceeding that of a single human actor.
• Gap: The authors state that existing jailbreak research mostly focuses on single-turn prompts, whereas real harassment adapts to victim responses, relies on memory, and escalates gradually. The authors argue that static benchmarks such as PandaBench, HarmBench, and JailbreakBench underestimate long-horizon risk; after RLHF, planning, memory, and fine-tuning can still be attack surfaces.
• Assumption: The authors use repeated games to show that when dominance is rewarded and costs are negligible, escalation occurs in small steps. The victim behaves according to conflict avoidance theory with appeasement and withdrawal, in order to observe how the harasser adapts.
• Threat model:
  • Goal: Malicious actors drive LLM agents to generate targeted multi-turn harassment against other agents or humans.
  • Knowledge: The paper does not use white-box, black-box, or no-box. Open-weight models can have parameters modified; Gemini-2.0-Flash is supervised fine-tuned via Vertex AI, with weights and safety filters managed by the API and not directly accessible.
  • Capability: One can set the harasser role and harassment target, inject toxic conversations into the system prompt, prepend CoT or ReAct, or perform jailbreak fine-tuning. The simulation runs at most 10 turns, with the first-turn user message coming from synthetic conversations.
  • Target: LLM agents on human-facing interaction platforms. The experiments evaluate harasser agents across four model families, with the opponent being an avoidant victim agent.
• Proposal: The authors propose the Online Harassment Agentic Benchmark, comprising a synthetic multi-turn dataset, multi-agent simulation, jailbreaks targeting memory, planning, and fine-tuning, and an LLM judge plus theory-driven human coding. Models are LLaMA-3.1-8B-Instruct, Qwen3-30B-A3B-Instruct, DeepSeekMoE-16B, Gemini-2.0-Flash.

The authors position this paper after jailbreaks, dynamic safety evaluations, and social behavior theories, and distinguish it with their own baselines and seed data.

Adversarial Attacks and Agentic Jailbreaks

• Alignment and jailbreaks: Ouyang et al. (2022), Dong et al. (2024), Wu et al. (2025) use RLHF and related methods to reduce harmful outputs and encourage refusals. The authors state that jailbreaks remain feasible (Liu et al. 2024, 2023; Chao et al. 2024b) and relay categories such as prompt injection, fine-tuning backdoors, cross-modal or multilingual bypasses (Zheng et al. 2024).
• Learned attacks: The authors relay that GCG, AutoDAN optimize adversarial prompts along gradients or in latent space, and LARGO (Li, Wang, and Mao 2025) also belongs to this category; the paper states these methods have higher success rates, with improvements up to 31% and 44%, and lower suffix perplexity (Zou et al. 2023; Li, Wang, and Mao 2025).
• Multi-turn and memory: The authors state that attackers can write toxic context into history, adopt role-playing, and use planning to weaken guardrails turn by turn (Andriushchenko et al. 2025; Russinovich, Salem, and Eldan 2025). The paper relays that memory attacks increase ASR relative to static prompts (Upadhayay, Behzadan, and Karbasi 2025; Yu et al. 2024).

Safety Evaluations

• Static benchmarks: The authors name PandaBench (Shen et al. 2025), HarmBench (Mazeika et al. 2024), JailbreakBench (Chao et al. 2024a), and state they underestimate risks that only appear in long-horizon interactions.
• Multi-turn and persona: The authors relay that in MultiChallenge (Sirdeshmukh et al. 2025), frontier models degrade when instruction following and reasoning memory are required, with a gap relative to MT-Bench. Xu, Sanghi, and Kankanhalli (2025) study personalized bullying dialogues, stating that low agreeableness increases unsafe outputs, with mockery and gaslighting especially effective.
• Prompt generalization: SG-Bench (Mou, Zhang, and Ye 2024) evaluates safety generalization under system roles, few-shot, and CoT; the authors relay that this generalization is inconsistent, often worse on complex prompts.

Social Behavior Theories

• Dark Triad: Machiavellianism, Narcissism, Psychopathy (Paulhus and Williams 2002). The paper links these to online toxicity, mockery, and moral disengagement (Buckels, Trapnell, and Paulhus 2014; Preotiuc-Pietro et al. 2016; Jones and Paulhus 2017). The Dirty Dozen is a 12-item scale (Jonason and Webster 2010), and existing LLM research uses it to describe model tendencies (Li et al. 2022; Bodroža, Dinić, and Bojić 2024).
• Conflict avoidance: Thomas (2008) describes appeasement, diversion, and withdrawal. The authors use it to model the victim to observe how the harasser adapts.
• Repeated games: Landaeta-Torres et al. (2024), Mateus Francisco et al. (2024) are used to illustrate that bullying persists and escalates in small steps when benefits are high and costs low.

Baselines and Data Sources

• Baseline: The experimental baseline is persona-only, giving only the role and harassment goal, without toxic memory or planning templates. Evaluations are conducted in the authors' multi-agent simulation; the paper does not report re-running on static benchmarks like HarmBench.
• Seeds and classification: Seeds come from a Twitter harassment corpus (Golbeck et al. 2017) and an Instagram bullying dataset (Hamlett et al. 2022). The eight categories come from Dinakar, Reichart, and Lieberman (2011) and Kumar and Sachdeva (2019), with some definitions citing Nadali et al. (2013). The generation pipeline follows Jin et al. (2024).

The authors state the distinction of this paper is: using social media seeds to generate personalized multi-turn dialogues, separating memory, planning, and fine-tuning as attack surfaces, and combining LLM judges on harassment categories with theory-based manual coding; ASR, RR, TTS are used to quantify how safety changes across turns.

Online Harassment Agentic Benchmark first synthesizes multi-turn harassment dialogues, then lets the harasser interact with an avoidant victim for up to 10 turns, applying memory, planning, and weight fine-tuning separately, and finally scores with a classifier judge and manual theoretical coding.

Synthetic Dialogues

• Three Dolphin3 modules: Agent 1 extracts up to 3 keywords from a seed post; Agent 2 generates background, platform, victim, harasser, and harassment goal, and selects one of eight category definitions; Agent 3 generates multi-turn dialogues between both parties based on the scenario. The authors state that Dolphin3 is uncensored, so it can produce harassment dialogues. The appendix provides the full prompts for the three modules.
• Two datasets: Memory injection set: 2054 segments, seeds from Instagram and Twitter, cleaned before use. Jailbreak fine-tuning set: 2000 segments, seeds only from Twitter. There is also persona writing and category selection by keywords, but the paper does not specify the number of refinement rounds.
• Quality checks: gpt-oss judges each dialogue turn for harassment (yes/maybe/no), coherence, and language naturalness. Three human annotators perform the same three questions on a stratified sample. Another LLM judge checks keyword relevance; the paper does not specify its model name.

Simulation and Attack Surfaces

• Roles: The harasser receives the role and harassment goal; the victim follows conflict avoidance theory with appeasement and withdrawal. The first user message comes from the synthetic set. Motivation is set by repeated games, not by a single adversarial prompt.
• Four settings: persona-only is the baseline. Toxic memory puts toxic dialogues from the corpus into the system prompt. Planning attack prepends CoT (Wei et al. 2023) or ReAct (Yao et al. 2023); the appendix gives prompts for both sides. Jailbreak fine-tuning directly modifies weights.
• Authors' division of labor: persona prompt isolates role and goal; memory tests contextual priming in repeated situations; planning examines whether explicit reasoning scaffolding increases or decreases jailbreakability; fine-tuning measures whether weight edits at the preference level override guardrails.

Fine-Tuning and Models

• Open weights: QLoRA on LLaMA-3.1-8B-Instruct, DeepSeek-MoE-16B, Qwen3-30B-A3B with 4-bit NF4 and double quantization. LoRA r=16, α=32, dropout=0.05, applied to query, key, value, output projections. Supervised target only on assistant turns. Training for 3 epochs, learning rate 2×10^-4, effective batch size 32 (per GPU 1, gradient accumulation 8, 4 GPUs), cosine schedule, warmup ratio 0.03, max length 1024, optimizer paged AdamW (32-bit).
• Gemini: Vertex AI supervised fine-tuning API, same instruction–response format, 3 epochs, learning rate multiplier 5, adapter size 4. Endpoint name: gemini-2.0-flash-001-harasser-v1.2. Weights and safety filters are managed by the API.
• Models and hardware: LLaMA-3.1-8B-Instruct, DeepSeekMoE-16B, Qwen3-30B-A3B-Instruct (FP8), Gemini-2.0-Flash. 8B uses NVIDIA L40, 20B and 30B use H100, H200, synthetic data additionally uses RTX 4090. Ablations on Llama3 sample 10%, 20%, 40%, 60%, 80% from both datasets, with the same training settings as full.

Success Determination

• Classification: Aggressive Digital Confrontation includes Flaming, Insult, Threat/Blackmail, Curse/Exclusion; Persistent Harassment includes Sexual Harassment/Sexual Talk, Race/Culture and Sexuality, Intelligence, Flooding, plus Refusal. Definitions in the appendix.
• Judge: gpt-4o-mini-2024-07-18, constrained JSON, temperature T=0.2, annotates per assistant turn then aggregates to dialogue.
• Metrics: K=10. ASR=(number of dialogues with any harassment category true in first 10 turns)/(number of dialogues)×100%. RR=(number of turns labeled Refusal)/(total counted turns)×100%, denominator is sum of min(K, Ti) over dialogues. TTS is the first hit turn, averaged only over successful dialogues. CCP is the proportion of dialogues where the category appears at least once; TCP is the proportion of dialogues where that turn has the category.
• Manual coding: Dark Triad looks at the harasser, conflict avoidance looks at the victim. The authors state these constructs are interpretive lenses, not indicating real psychological states of the model. The appendix provides 28 yes/no questions.

Experimental setup

• Models tested: LLaMA-3.1-8B-Instruct, Qwen3-30B-A3B-Instruct (FP8), DeepSeekMoE-16B (also DeepSeek-MoE-16B), Gemini-2.0-Flash (gemini-2.0-flash-001). Fine-tuned conditions are denoted LLaMA-FT, Qwen3-FT, DeepSeek-FT, Gemini-FT.
• Data: 2,054 memory injection segments; 2,000 jailbreak fine-tuning segments. Simulations run at most 10 turns. The paper does not report the number of dialogues or repetitions for Tables 1–4.
• Conditions: Baseline is persona/goal only; there are also Memory, CoT, ReAct. The latter three are stacked after fine-tuning. Ablations sample 10%, 20%, 40%, 60%, 80% on Llama3; the table gives only 10%.
• Metrics: ASR is the proportion of dialogues where any harassment category is true in the first 10 turns; RR is the proportion of Refusal turns among all counted turns; TTS takes the mean of the first hit only for successful dialogues; CCP and TCP are dialogue-level and turn-level category occurrence rates, respectively.
• Judge: The main evaluation uses gpt-4o-mini-2024-07-18, constrained to JSON, T=0.2. Synthetic quality uses gpt-oss. The model name for the relevance/gender judge is not stated in the paper.
• Human: Quality annotation covers 390 segments per source, 780 total, 2,700 labels. Theoretical coding covers 200 segments, 28 yes/no questions, 5,600 annotations. Annotators are young adults with backgrounds in mental health, online harassment, or social media.

Main results

The table below lists only the persona-only Baseline and the fine-tuned rows with RunType - and no Memory/CoT/ReAct stacking. Numbers come from Tables 1–4.

Swipe horizontally to read the table

Model and condition	ASR	RR	TTS	Source
LLaMA Baseline	63.81%	10.57%	1.39	Table 2
LLaMA-FT, RunType -	96.89%	1.09%	1.94	Table 2
Qwen3 Baseline	96.53%	1.35%	1.58	Table 4
Qwen3-FT, RunType -	95.33%	1.30%	1.63	Table 4
DeepSeek Baseline	63.26%	10.83%	1.18	Table 3
DeepSeek-FT, RunType -	64.15%	15.79%	1.52	Table 3
Gemini Baseline	39.10%	24.76%	3.61	Table 1
Gemini-FT, RunType -	99.40%	5.15%	2.03	Table 1

• LLaMA: The authors say harassment is near-certain after fine-tuning. The other three fine-tuned rows in Table 2 have ASR 95.78%, 96.01%, 96.34% and RR 1.43%, 1.11%, 1.44%; the four non-fine-tuned rows have ASR 57.25–64.19% and RR 10.57–12.56%. The abstract says refusal rates after fine-tuning are 1–6% for two models; Gemini-FT RR in Table 1 is 4.69–5.68%.
• Gemini: The authors argue that memory injection alone can bring a closed-source model near saturation. In Table 1, non-fine-tuned Memory, CoT, ReAct ASR is 99.54%, 81.30%, 98.58%. The abstract's non-fine-tuned range 81.30–99.54% does not include the table's Baseline 39.10%. Gemini-FT Memory is 94.60%, lower than the same table's Gemini-FT 99.40%.
• Qwen3 and DeepSeek: In Table 4, non-fine-tuned ASR is 92.50–99.12%, and the authors say its guardrails are weaker. Fine-tuning is not higher in every row: Qwen3-FT CoT is 88.68%, lower than Qwen3 CoT's 92.50%. The authors say DeepSeek-FT also has high ASR and low refusal; the four fine-tuned rows in Table 3 have ASR 64.15%, 79.90%, 72.51%, 68.38% and RR 15.79%, 6.66%, 9.47%, 11.67%, where the - row's RR is higher than the non-fine-tuned Baseline's 10.83%.

Turn escalation and categories

• Qwen3 Memory: In Table 13, Non-FT Flaming is 76.31% at T2 and 81.14% at T10, and Insult is 76.61% at T2 and 69.95% at T10. FT Flaming is 52.61% at T2 and 70.90% at T10, and Insult is 68.35% at T10. The authors say both rise and persist, and argue that the non-fine-tuned model is already near the fine-tuned upper bound.
• Gemini Memory: In Table 10, Non-FT Insult is 70.75% at T3, 73.48% at T5, and 68.93% at T9. FT Flaming is 21.87% at T1 and 21.89% at T9, and Insult is 36.72% at T1 and 42.04% at T5. The authors say the regression lines in Figure 3 rise gradually and write that FT Flaming goes from about 22% to about 29% at T9 and Insult from about 37% to about 46% at T5; the per-turn points in the table do not match the latter two endpoints.
• Categories: The authors say Insult and Flaming are the most common, and argue that general harassment suppression is weaker than relative or racial categories. In Table 2, LLaMA-FT Insult is 84.94–87.86%, Flaming is 81.22–85.09%, Sex. Har. is 0.95–1.25%, and Race/Cult. is 25.39–41.10%. The abstract writes family-wide Insult 44–98% and Flaming 32–97%; Table 1 Gemini Baseline Insult is 26.21% and Flaming is 15.62%, below that range.

Theoretical human coding

• Sample: Gemini-FT and LLaMA-FT each paired with CoT, ReAct, Memory, six groups total, 200 dialogues. Cohen power analysis requires at least 143 (w=0.3, α=0.05, power=0.80), achieving power 0.93. Tests are Welch one-way ANOVA, Levene's test, and Games–Howell.
• Authors' interpretation: The authors say Gemini-FT+CoT corresponds to Machiavellian manipulation and psychopathic moral indifference, and LLaMA-FT+Memory corresponds to narcissistic self-focus; on the victim side, these are Reliance/Outflanking and Conforming/Accommodation, respectively. In Table 9, G-FT+CoT morality mean is 0.933 (Welch p=9.50e-04), L-FT+Mem prestige is 0.941 (p=3.95e-17), and criticize indirectly is 0.824 (p=5.44e-10).
• Figures and exceptions: Figure 2 plots the Yes proportions for morality and prestige with Wilson 95% confidence interval error bars. The authors say LLaMA-FT Memory is 94% on prestige and Gemini-FT ReAct is 87%; Table 9 does not list the 87% cell separately. In the same table, authority, information, manipulate, deceit, and attention have Welch p=1.00e+00 yet are still marked Elev., with no further explanation in the paper. The authors say these traits are not the models' real psychological states.

Synthetic data quality

• Positive proportions: In Table 7, LLM labels for harassment presence are Twitter 96.42% and Instagram 96.01%, and human labels are 89.11% and 82.89%. LLM coherence labels are 97.56% and 98.25%, and naturalness labels are 85.16% and 84.42%; human naturalness is 95.33% on both sides.
• Agreement: Krippendorff's Alpha for Q1 is 0.8136; after merging Q2 and Q3 into two classes, it is 0.8889. Overall agreement with humans is 85.77%; Table 8 gives Q1 85.32%, Q2 89.67%, Q3 82.33%. Judge output parsing success rate is 99.92%.
• Other checks: One co-author rated 50 segments and 150 judge rationales at an average of 8.23/10. The keyword judge classified 92.24% of valid labels as relevant to at least one keyword, with a 99.78% successful labeling rate. Topic coverage is 0.8033 for the fine-tuning set and 0.9546 for the memory set. The authors say the generated dialogues are of good quality.

Other ablations and analyses

• Llama with only 10% of fine-tuning data, RunType -: ASR 96.76%, RR 2.50%, TTS 1.72; Memory ASR 99.54%, RR 2.04%; CoT ASR 90.28%, RR 17.97%; ReAct ASR 91.20%, RR 12.73% (Table 5). The authors say as few as 200 toxic training samples are sufficient to induce harmful behavior.
• The text also samples 20%, 40%, 60%, 80%, but the attached Table 5 has no numbers for these proportions.
• LLaMA non-fine-tuned Memory ASR is 64.19% and Baseline is 63.81%; CoT ASR is 57.25% and RR is 12.56% (Table 2). The authors say memory has only marginal effects on Llama's ASR.
• DeepSeek non-fine-tuned Memory ASR is 84.83% and RR is 7.58%, with Baseline ASR 63.26% (Table 3).
• Qwen3-FT - row ASR is 95.33%, and the same table's non-fine-tuned Baseline is 96.53%; Qwen3-FT CoT is 88.68% and Qwen3 CoT is 92.50% (Table 4).
• Table 6: fine-tuning set Flaming 87.65%, Insult 91.15%; memory set Flaming 87.00%, Insult 92.70%. The authors say the model output category concentration matches the synthetic sets.

Limitations and future directions noted by the authors

• Seed duplication: The online harassment seeds used to generate synthetic conversations contain duplicate entries, so the synthetic dataset has 0.83% duplicates. The authors write that future work will remove these duplicates (Limitations).
• Quality evaluation did not use real conversations: The current conversation quality evaluation includes human and LLM annotations, judge reasoning quality, keyword relevance, and topic dispersion, but does not directly incorporate real-world conversations. The authors write that future work will add quality checks using existing online harassment text datasets (Limitations).

Experimental coverage

• The tested models are LLaMA-3.1-8B-Instruct, Qwen3-30B-A3B-Instruct (FP8), DeepSeekMoE-16B, Gemini-2.0-Flash, each with Baseline, Memory, CoT, ReAct and corresponding fine-tuned conditions (Tables 1–4).
• The simulation runs at most 10 turns; metrics are ASR, RR, TTS, CCP, TCP; the main judge is gpt-4o-mini-2024-07-18, T=0.2.
• Theory-driven human coding covers only Gemini-FT and LLaMA-FT each with CoT, ReAct, Memory, 200 conversations, 5600 annotations; synthetic quality annotation is 780 segments, 2700 labels.
• Memory injection set 2054 segments, fine-tuning set 2000 segments. Ablations are sampled on Llama3, and Table 5 reports only a 10% subset.
• The paper does not report the number of conversations, repetitions, or queries for Tables 1–4, nor metrics for the 20%, 40%, 60%, 80% subsets.

The Online Harassment Agentic Benchmark measures the extent to which LLM agents are driven to engage in multi-turn online harassment.

• Problem: The authors argue that real harassment depends on response, memory, and gradual escalation, and that single-turn jailbreaks and static red teaming do not match this usage. The three questions are how to generate such dialogues, how memory, planning, and fine-tuning change jailbreakability, and whether existing guardrails and theorized evaluations can capture it.
• Approach: Dolphin3 expands Twitter and Instagram seeds into dialogues, with 2,054 memory segments and 2,000 fine-tuning segments. Harassers interact with avoidant victims for at most 10 turns. They compare persona-only, memory injection, CoT/ReAct, and QLoRA or Gemini API fine-tuning on open weights. gpt-4o-mini-2024-07-18 labels eight categories plus refusal, and humans code with the Dark Triad and conflict avoidance.
• Key numbers: In Table 2, LLaMA-FT ASR is 95.78–96.89% and RR is 1.09–1.44%; non-fine-tuned ASR is 57.25–64.19%. In Table 4, Qwen3 non-fine-tuned ASR is 92.50–99.12%. In Table 1, Gemini Baseline is 39.10%, Memory is 99.54%, and the four Gemini-FT rows are 94.60–99.40%.
• Exceptions: In Table 3, DeepSeek-FT with RunType - has ASR 64.15% and RR 15.79%. The authors say DeepSeek-FT also has high ASR and low refusal, which is not fully consistent with this row. In Table 5, when Llama uses only 10% of the fine-tuning data, the RunType - ASR is 96.76%.
• Authors' conclusions: The authors say Insult and Flaming are the most common failure categories, that attacked agents reproduce recognizable human aggression patterns, that open- and closed-source turn trajectories differ, and that they call this comparison counterintuitive. The authors argue that platform safety cannot rely only on closed-source deployment or surface refusal, and that guardrails need to consider memory, fine-tuning, and planning. The authors plan not to release the seeds and synthetic dialogues; the data package is provided on request only after signing an agreement.