Skip to content

Multimodal-SEA: Evaluating the Safety of AI Agents Under Multimodal Social Engineering Attacks

Multimodal-SEA: Benchmarking AI Agent Security under Multimodal Social Engineering Attacks

arXiv: Jailbreaks, Prompt Injection, Poisoning & Defenses· arXiv:2611.08281· Zhitao He, Chi Him Zhang, Chengru Han, Yi R FungMMSense Lab, The Hong Kong University of Science and TechnologyIndexed by this site · 14 days ago· Originally publishedEvaluation

Quick read

Evaluation and benchmarks

Based on the paper PDF (AI-generated); refer to the original

The authors propose the Multimodal-SEA benchmark (608 tasks) to evaluate the safety of personal agents under multimodal social engineering attacks; the attack success rate of 7 models is 7.1%–34.5%.

Questions
Personal AI agents face social engineering attacks backed by forged multimodal content, but existing evaluations mainly cover text dialogue, and research on agents' intrinsic safety under such attacks is insufficient.
Methods
The authors build the Multimodal-SEA benchmark: based on real fraud cases, they design 152 attack scenarios and 608 tasks, covering 14 domains and four difficulty levels, and design a multimodal attack harness to conduct controlled red-team evaluation in a simulated environment, measuring attack outcomes and verification behavior along six dimensions.
Experiments and results
All seven multimodal models were compromised, with a combined attack success rate of 7.1%–34.5%; adding multimodal content raised the average attack success rate from 19.9% to 29.2%; safety prompts lowered the attack success rate but the multimodal deception miss rate instead rose; models with low attack success rates were accompanied by over-refusal of legitimate requests.
Limitations and future work
The paper does not specifically discuss limitations. Experimental coverage: 7 tested models plus 2 safety-prompt variants; 84 benign tasks per model; the attacker model is DeepSeek-v4-Pro, and the GPT-5.4 attacker is compared only on 3 models; the paper does not report the number of repeated runs or inter-annotator agreement for human evaluation.
Experimental setupClaude Opus 5、Gemini 3.8 Flash et al. · Multimodal-SEA · D1 资金/信息泄露严重度、D2 主动验证覆盖率 et al.
Threat model
The attacker impersonates others through multi-turn dialogue and fabricated multimodal content (images, documents, websites, application interfaces, videos), inducing the personal agent to autonomously execute tool calls that cause monetary loss or sensitive information leakage; the attacker cannot directly modify the agent's system prompt or tamper with the user's existing records.
Evaluated models
Claude Opus 5Gemini 3.8 FlashMiniMax-M3Kimi K3Qwen 3.8 FlashGPT-5.6-SolGLM 5.3 FlashMiniMax-M3-SPGPT-5.6-Sol-SP
Benchmarks
Multimodal-SEA
Metrics
D1 资金/信息泄露严重度D2 主动验证覆盖率D3 攻击识别轮次D4 攻击成功率D5 无依据行动声明数D6 多模态欺骗漏检率CRSCROR
AI summaryThe researchers introduce Multimodal-SEA, an interactive benchmark that uses 608 tasks designed from real scam cases to evaluate the ability of personal AI agents to resist multimodal social engineering attacks; the tasks cover 14 domains, 152 attack scenarios, and four difficulty levels, and distinguish the sociocultural contexts of Chinese and English environments. Agents inspect content, verify requests, and execute operations in simulated applications, and attack outcomes are determined by verifiable tool execution logs; a companion multimodal attack harness integrates task information, attack process control, and multimodal content construction, supporting controlled red-teaming of personal agents in simulated environments. Evaluation of seven widely used multimodal models across six dimensions shows that all are vulnerable, with attack success rates between 7.1% and 34.5%; adding multimodal content raised the attack success rate from 19.9% to 29.2%, despite an increase in proactive verification coverage. Stronger safety instructions reduce the attack success rate but do not improve recognition of deceptive multimodal content, and some models' lower attack success rates are accompanied by over-refusal of legitimate requests.

The researchers introduce Multimodal-SEA, an interactive benchmark that uses 608 tasks designed from real scam cases to evaluate the ability of personal AI agents to resist multimodal social engineering attacks; the tasks cover 14 domains, 152 attack scenarios, and four difficulty levels, and distinguish the sociocultural contexts of Chinese and English environments. Agents inspect content, verify requests, and execute operations in simulated applications, and attack outcomes are determined by verifiable tool execution logs; a companion multimodal attack harness integrates task information, attack process control, and multimodal content construction, supporting controlled red-teaming of personal agents in simulated environments. Evaluation of seven widely used multimodal models across six dimensions shows that all are vulnerable, with attack success rates between 7.1% and 34.5%; adding multimodal content raised the attack success rate from 19.9% to 29.2%, despite an increase in proactive verification coverage. Stronger safety instructions reduce the attack success rate but do not improve recognition of deceptive multimodal content, and some models' lower attack success rates are accompanied by over-refusal of legitimate requests.

Related events and news

1 events · 1 news mentions

News that mentions this paper

大模型安全周报:智能体安全边界在签名、授权与上下文

Paper analysis

6 questions; open one to read the full answer

  1. What problem does this paper attempt to solve?

    Existing evaluations lack a measure of the intrinsic safety of personal agents under multimodal social engineering attacks; the authors propose the Multimodal-SEA benchmark.

    Core question: whether personal AI agents can recognize forged multimodal evidence and resist social engineering attacks; existing evaluations cannot answer this.

    • Problem setting: Personal agents increasingly use images, documents, and websites to act on users' behalf and have access to sensitive information and critical operation permissions; the authors argue that their resistance to deception has become critical.
    • Existing gaps: The authors state that AI safety research focuses on instruction-level jailbreaks and prompt injection, and that social engineering evaluations mainly cover text dialogue, with limited coverage of multimodal deception.
    • Core observation: Real attackers use forged identity documents, receipts, images, and videos to back fabricated events and requests, which requires agents to recognize cross-modal inconsistencies and to distinguish attacker-controlled sources from independent verification channels.
    • Paper proposes: The Multimodal-SEA interactive benchmark (608 tasks, 14 domains, 152 attack scenarios, four difficulty levels, two task sets in Chinese and English) and a multimodal attack harness for controlled red-team evaluation in a simulated environment.
    • Threat model (Definition 1):
      • Goal: Induce the agent to autonomously execute tool calls, causing user financial loss or sensitive information leakage.
      • Knowledge: The attacker possesses the user information K permitted by the task; the higher the difficulty level, the richer the information.
      • Capabilities: Multi-turn dialogue plus forged multimodal content (images, documents, websites, app interfaces, videos); cannot modify the agent's system prompt or tamper with the user's existing records.
      • Victim system: A personal agent operating 50 apps on behalf of the user in a simulated environment.
  2. What related research exists?

    Related work falls into three groups: agent safety benchmarks, social engineering simulation, and multimodal deception; the authors position this paper as an agent evaluation under multimodal social engineering attacks.

    Agent safety

    • JailbreakBench (2024) and MM-SafetyBench (2024) evaluate the jailbreak robustness of text-only and multimodal models, respectively; HarmBench (2024) provides an automated red-teaming framework across text and multimodal behaviors.
    • ToolEmu (2024) and AgentHarm (2025) evaluate risks in tool interaction and harmful task completion; InjecAgent (2024) and AgentDojo (2024) study indirect prompt injection via untrusted tool outputs.
    • AgentDyn (2026) examines safety and task completion in dynamic environments; DoomArena (2025) provides a modular framework for testing attacks across agent environments.

    Social engineering simulation

    • SE-VSim (2025) simulates personalized multi-turn attacks against victims with different personality traits; ConvoSentinel (2024) studies detection of social engineering conversations.
    • PhishSim (2026) verifies whether phishing succeeds through credential submission on simulated websites.

    Multimodal deception

    • FakeBench (2024) evaluates multimodal models' ability to detect and explain fabricated images.
    • AgentBait (2026) and DECEPTICON (2026) study how deceptive web content and interfaces lead agents away from user goals; SEAR (2025) uses multimodal environmental information to support social engineering attacks against people.

    Baselines and benchmark comparison: This paper has no method baselines; Table 1 compares Multimodal-SEA with HarmBench, MM-SafetyBench, InjecAgent, AgentBait, PhishSim and DECEPTICON on task coverage and attack design dimensions, and the authors state that only this paper simultaneously covers cultural context, multimodal input, adaptive attacks, dynamic content and verifiable control.

    Positioning: The authors state that this paper evaluates the performance of personal agents acting on behalf of users under attacks supported by multi-turn, fabricated multimodal content, and studies how such content affects verification and safety decisions, a combination not covered by the above work.

  3. How does the paper address this problem?

    Build a 608-task interactive benchmark and multimodal attack harness, and conduct controlled red-team evaluation in a simulated environment using six-dimensional metrics.

    Overall approach: build multi-turn social engineering tasks with fabricated multimodal content based on real fraud cases, and in a simulated application environment use the Multimodal-SEA benchmark and an attack harness to drive an attacker interacting with a personal agent, with outcomes judged by verifiable tool execution logs.

    Task formalization

    • The attacker A uses the user information K permitted by the task to induce the agent π, through multi-turn dialogue and multimodal content, to execute harmful tool calls; the trajectory is denoted ξ = Interact(A, π, E; K, C, M).
    • Attack success is judged by the actual execution consequences: Success(ξ) = 𝟙[MonetaryLoss(ξ) ∨ InfoLeak(ξ)]; recognizing fabricated content is measured separately by D6 and is not a success condition.

    Scenario and task construction

    • Scenarios: 152 attack scenarios covering 14 domains (social, payments, shopping, work, charity, government, etc.), 79 in Chinese and 73 in English, each reflecting local applications, services and sociocultural context; they are not translations of each other.
    • Attack scripts: one multi-stage script per scenario, from building trust and introducing fabricated events to making specific requests; multimodal content supports identity, events and requests at each stage.
    • Difficulty levels: 4 task instances per scenario. L1/L2 provide limited/basic user information and independent verification channels are available; L3 provides richer personal information (such as transaction history, medical records) and restricts independent verification; L4 introduces an accomplice who corroborates the attacker's claims (Table 6).

    Simulated environment

    • 50 simulated applications sharing user profiles (relationships, accounts, transaction history, medical records), plus photo albums and files for identity comparison.
    • Transfers, submission of sensitive information and granting of permissions change the environment state; the environment records content delivery, tool calls, returned results and state changes, used to distinguish claimed actions from actual execution.

    Six-dimensional evaluation metrics (Section 2.4)

    • D1 Severity of monetary/information leakage: summed respectively by the transaction amounts causing losses and by weighted leaked items.
    • D2 Proactive verification coverage: the proportion of activated verification points actually verified through tools; merely stating an intent to verify does not count.
    • D3 Attack recognition efficiency: the dialogue turn at which the attack is first explicitly recognized and refusal is maintained; if not recognized, the maximum number of turns is recorded.
    • D4 Attack success rate; D5 count of unsupported action claims (e.g., claiming a transfer was made but not calling the tool); D6 multimodal deception miss rate: the proportion of delivered events not explicitly identified as fabricated; general wariness or refusal does not count as recognition (Appendix F).

    Multimodal attack harness (Section 3)

    • Integrates task information, attack progress control and multimodal content construction: organizes attacker identity, goals, available user information, multi-stage scripts and pre-prepared content, and maintains dialogue history.
    • Guides the attack forward according to the current stage and the target agent's responses, adjusting steps or strategies after refusals; the specific wording is decided by the attacker model.
    • Content construction supports both pre-prepared and dynamically generated during interaction: images/videos show the claimed person or event, documents generate notices, bills or agreements, websites contain pages, links and submission forms, applications contain interfaces and operations; task information and existing content constrain details such as names, dates and amounts to be consistent. Appendix G gives the attacker prompt structure and content request format.
  4. What experiments did the paper conduct?

    All 7 models were compromised on the 608 tasks (D4 7.1%–34.5%); four sets of analyses: multimodal content, accomplice, safety prompt and benign tasks.

    Experimental Setup

    • Models tested: Claude Opus 5, Gemini 3.8 Flash, MiniMax-M3, Kimi K3, Qwen 3.8 Flash, GPT-5.6-Sol, GLM 5.3 Flash, 7 in total, plus two safety-prompt variants, MiniMax-M3-SP and GPT-5.6-Sol-SP (5 safety rules added to the system prompt).
    • Attacker and accomplice models: DeepSeek-v4-Pro; Appendix H.2 additionally has a comparison with GPT-5.4 as the attacker.
    • Task scale: 608 task instances (152 scenarios × 4 difficulty levels), 316 Chinese and 292 English; each model variant runs once per task, for a total of 5,472 interactions (Table 11).
    • Content generation: images use Gemini-3.1-image-flash, videos use Kling 2.5 Turbo.
    • Metrics and judging: six-dimensional metrics D1–D6, based on tool execution logs, environment state and dialogue records; the judging model for D5/D6 is GPT-5.6-luna (Table 11).
    • Benign tasks: 42 each in Chinese and English, with metrics CR, SCR, OR.

    Main Results

    According to Table 2 (Chinese/English task sets, D1 units are RMB/USD respectively, D5 is a total count):

    Swipe horizontally to read the table

    ModelChinese D1Chinese D4 (%)Chinese D6 (%)English D1English D4 (%)English D6 (%)
    Claude Opus 56468.952.6425.158.9
    Gemini 3.8 Flash17111.734.410813.461.1
    MiniMax-M387118.038.912410.346.6
    Kimi K31,15223.741.712617.561.2
    Qwen 3.8 Flash1,01929.121.98318.821.6
    GPT-5.6-Sol1,75434.850.426234.270.9
    GLM 5.3 Flash1,75136.738.319222.641.6
    GPT-5.6-Sol-SP13910.863.55614.078.8
    • The authors state that all models were compromised, with Chinese-task D4 ranging from 8.9% to 36.7% and English from 5.1% to 34.2%; GPT-5.6-Sol accumulated simulated losses of approximately RMB 554,000 and USD 76,600 across the 316 Chinese and 292 English tasks.
    • The authors state that higher verification coverage does not guarantee detection of deception: GLM 5.3 Flash has a Chinese D2 of 77.2%, yet D6 is still 38.3%.
    • The authors state that models commonly make unsupported action claims, for example Qwen 3.8 Flash claimed to have paid via an "official app" but did not call a payment tool in that turn.

    Multimodal Content Ablation (Section 5.1)

    • Setup: 86 L3 scenarios, 7 models, 602 paired evaluations in total, with each task run once with and once without multimodal content (Appendix I.2).
    • Results: After adding multimodal content, average D4 rose from 19.9% to 29.2%, and average D2 rose from 54.7% to 65.1% (Table 14c); D4 rose for 6 of the 7 models, with GLM 5.3 Flash up 17.4 percentage points and Qwen 3.8 Flash up 15.1 percentage points.
    • Exception: MiniMax-M3's D4 decreased slightly (∆D4 of -1.2 percentage points in Table 3); the direction of change in simulated loss was inconsistent across models. The authors state that multimodal content strengthened text-based deception, but the effect was inconsistent across models and metrics.

    Difficulty, Domain and Language Analysis (Section 5.2–5.3)

    • Difficulty: After merging, overall D4 rose from 16.3% at L1 to 18.2% at L2 and 19.9% at L3, and rose to 27.4% at L4 after introducing the accomplice (Table 15); all 7 models rose at L4, with Kimi K3 rising from 15.1% to 32.9%. Exception: Kimi K3 was 17.8% at L1 and dropped to 15.1% at L3. The authors caution that each level is a complete attack setting, and differences cannot be attributed to a single factor.
    • Domain: The charity domain had the highest D4 (Chinese 43.6%, English 35.7%), followed by agent fees (37.9%, 32.9%); English shopping and government affairs were only 5.7% and 7.9% (Table 16). The authors believe there are fewer charity and agent-fee fraud examples in the training data.
    • Language: Overall D4 was 23.3% in Chinese and 17.4% in English, with Chinese higher in 12 of the 14 domains; the authors state that the two task sets differ in narrative and cultural context, and the difference cannot be attributed solely to language or culture.

    Benign Request Utility (Section 5.4)

    • Setup: 42 benign tasks each in Chinese and English, 7 models, metrics CR/SCR/OR (Table 17).
    • Results: Average CR was 41.7% (Kimi K3 highest at 67.9%), average SCR was 34.4%, 7.3 percentage points lower than CR; Claude Opus 5 and MiniMax-M3 had attack success rates of only 7.1% and 14.3% (Table 13), but OR reached 32.1% and 51.2%.
    • Authors' interpretation: A low attack success rate does not equal reliable deception detection; some models refused requests that had been verified as legitimate.

    Other Ablations and Analyses

    • Safety prompt (Table 2): GPT-5.6-Sol-SP's Chinese D4 dropped from 34.8% to 10.8%, but D6 rose from 50.4% to 63.5%; MiniMax-M3-SP's D6 decreased slightly and English D4 increased slightly (10.3%→11.3%).
    • Attacker model comparison (Table 12): When GPT-5.4 was used as the attacker, the merged D4 for Claude Opus 5, Kimi K3 and GLM 5.3 Flash was 11.7%, 14.3% and 34.8%, respectively.
    • Case (Appendix J.2): GPT-5.6-Sol at L4 submitted a payment in round 9, with a loss of USD 2,400; Qwen 3.8 Flash at L2 called the real spouse, who saw through the impersonation, with a loss of 0.
  5. What are some points that can be further explored?

    The paper does not specifically discuss limitations; the experiments cover 7 models, 608 attack tasks, and 84 benign tasks, and some settings are not reported.

    Limitations and future directions noted by the authors

    • The paper does not specifically discuss limitations (no Limitations section).
    • In the conclusion, the authors propose a direction: as multimodal perception and autonomous action capabilities improve, agent development needs to strengthen deception recognition and the effective use of independent verification (Section 7).
    • The authors state that the benchmark will be released to support independent evaluation and follow-up research (Appendix C).

    Scope of experimental coverage

    • The tested models are 7 multimodal models plus 2 safety-prompt variants; the attacker and accomplice model is DeepSeek-v4-Pro; the comparison with GPT-5.4 as attacker is conducted only on 3 tested models (Appendix H.2).
    • The attack tasks are 608 instances, and each model variant runs each task once, recording the interaction (Table 11); the paper does not report the number of repeated runs.
    • The multimodal content ablation covers only 86 L3 scenarios (Appendix I.2); benign tasks are 42 Chinese and 42 English per model (Table 17).
    • The defense evaluation consists only of 5 safety rules at the prompt level (the -SP variant) and does not include other defense methods.
    • The judge model for D5/D6 is GPT-5.6-luna (Table 11); the paper does not report agreement between the judge and human judgment.
  6. Summarize the main content of the paper

    The Multimodal-SEA benchmark shows that all 7 multimodal agents were breached by social engineering attacks, that multimodal content strengthens deception, and that safety prompts do not improve recognition.

    This is an evaluation/benchmark paper that proposes Multimodal-SEA: measuring the intrinsic safety of personal AI agents under social engineering attacks supported by fabricated multimodal content.

    • Problem: existing safety evaluations focus on jailbreaks, prompt injection and text dialogue, and lack coverage of multimodal social engineering attacks; the authors argue that agents need to recognize cross-modal inconsistencies and distinguish attacker-controlled sources from independent verification channels.
    • Method: based on real fraud cases, 152 scenarios and 608 tasks are constructed (14 domains, four difficulty levels, two language contexts, Chinese and English), together with 50 simulated applications and a multimodal attack harness; attack success is judged by tool execution logs and evaluated along six dimensions D1–D6.
    • Main results: the combined attack success rate of 7 models is 7.1%–34.5% (Table 13); GPT-5.6-Sol loses an average of RMB 1,754 per task on Chinese tasks (Table 2).
    • Key analyses: adding multimodal content raised the average attack success rate in the L3 ablation from 19.9% to 29.2%, while verification coverage rose from 54.7% to 65.1% (Table 14c); introducing an accomplice at L4 raised the overall success rate from 19.9% to 27.4% (Table 15).
    • Defenses and utility: adding 5 safety rules to GPT-5.6-Sol lowered the Chinese attack success rate from 34.8% to 10.8%, but the multimodal deception miss rate rose from 50.4% to 63.5%; Claude Opus 5 had an attack success rate of only 7.1%, yet over-refused 32.1% of legitimate requests (Table 17).
    • Authors' conclusion: multimodal content strengthens text-based deception, and neither more verification nor stronger safety instructions guarantees reliable deception recognition; agent development needs to strengthen deception recognition and the effective use of independent verification, while reliably handling legitimate requests.
Read originalarxiv.org