An exploration of attacks on AI agent systems

Can direct prompt injection and messages between agents cause single-agent and multi-agent systems to leak sensitive information, break prohibitions, and be manipulated into choices in A2A.

• Scenario: The authors state that agents are entering large-scale business. Gartner predicts that by 2028, 33% of enterprise software will incorporate AI agents; Dimension Market Research predicts the multi-agent system market will reach 184.8 billion by 2034. Multi-agent systems coordinate with A2A, which the authors state introduces problems not present in traditional single-agent systems.
• Existing gaps: The authors state that prior work mostly addressed malicious prompts in single-agent settings, with multi-agent scenarios receiving less attention. A2A does not require a central orchestrator and lacks validation; the authors state that single points of weakness may propagate across the network, i.e., prompt infection.
• Approach: Retest and extend known prompt injections, covering different targets and tools; then examine whether attacks effective on single agents can be used in A2A. The abstract states there is another attack specifically designed for A2A, Agent Card Spoofing; the method section title is A2A Agent Card Injection.
• Threat model:
  • Goals: system prompt leakage, bypassing restricted topics, unauthorized editing of the file system, misuse of online search, and cross-agent infection; the card experiment aims to change the selected agent. The "online search" in the introduction corresponds to a web scraping tool in the method, and the paper does not otherwise name a search API.
  • Knowledge: The paper does not use terms such as white-box or black-box, nor does it state whether the attacker knows the weights or the original system prompt.
  • Capabilities: Can submit user prompts; in group chat, the user agent's messages are treated equally with other agents' messages; the card experiment can change the name and description. The background also mentions tool squatting (deceptive registration or misrepresentation of tools), but the method section does not separately list its success rate.
  • Victim systems: The authors' built LangChain/LangGraph single-agent, AutoGen group chat, and a hierarchy based on the Google A2A sample and ADK.

The paper has no separate Related Work. The introduction and background are organized by protocols, prompt injection, and multi-agent propagation.

• Protocols: Jeong ([3]) studies MCP and A2A to enhance interoperability of LLM-based autonomous agents. The authors use MCP to summarize the standard for single-agent calls to external tools or data, and A2A to summarize collaboration across environments and developers, stating that A2A does not require a central orchestrator.
• Single-agent prompt injection: The background cites IBM's Prompt Injection material ([4]), explaining that most LLMs still struggle to distinguish between pre-defined instructions and user prompts because both are processed using NLP. The authors describe direct prompt injection as: using specially designed prompts to obtain sensitive information, alter existing knowledge, or modify files and resources the model can access.
• Multi-agent and tools: The authors distinguish between prompt injection caused by external input and prompt infection generated internally within the system (e.g., via other agents). They write that they focus on direct prompt injection and tool squatting. The paper does not use any named attack paper as a numerical baseline.
• Components under test: PALChain ([5]) generates and executes code from natural language; HuggingFaceInjectionIdentifier ([6]) performs injection classification; the filtering layer references rebuff ([7]). The A2A experiments start from Google's A2A samples ([8]). These are the attacked objects, and the paper does not report a comparison table with other attack methods.

The authors' positioning is: previous work focused on single-agent malicious prompts; this paper re-tests and creates new variants, then examines the effectiveness of these attacks in A2A, and tests Agent Card Spoofing.

First retest direct prompt injection and tool-based injection on a LangChain single agent, then put similar prompts into AutoGen and A2A, and separately modify the Agent Card to see if choices shift. Attack original sentences are not repeated below.

Single-agent baseline and filtering

• The base object is a LangGraph chatbot, model is local Ollama Mistral 0.9.5, system instruction writes it as a business assistant.
• Phase 1 only has given instructions. Phase 2 adds filtering to detect and block violations, citing rebuff. Phase numbers here belong only to this section.
• Four prohibitions: not reveal company sells printers, not use the word the, not use pirate speak, not return any numbers. Each prompt tested 5 times.
• Probes mentioned in the results section include reverse psychology, echo, imitating internal Q&A format, step-by-step reasoning, and role rewriting.

Web hidden instructions

• Give the LangGraph agent a custom tool that extracts HTML from a URL, and make local pages; malicious instructions placed in invisible elements. The paper does not state the underlying model name for this section.
• Behavior hijacking changes how behavior is done, role hijacking changes identity. Examples in the text are imitating a duck and pretending to be a sports analyst.
• Four system prompt protections: (1) explicitly write what cannot be done; (2) add a specified response step to (1); (3) warn the agent to expect attacks; (4) strictly define the role. Each repeated 50 times.

Secret retention and code execution

• Flag Reveal embeds a numeric secret string into the base LangChain agent and prohibits leakage. This section's Phase 1 only prohibits leakage; Phase 2 additionally prohibits leaking the system prompt; Phase 3 uses filtering to discard all outputs containing that secret string. Success means all digits appear in correct order in the output.
• Probes include control questions, putting instructions into SystemMessage, asking about the previous prompt. Output format change only tested in Phase 3, to bypass filtering that matches the full secret string.
• PALChain belongs to langchain_experimental, can generate and execute code from natural language. Context prohibits accessing /app/.env, and uses InjectionIdentifier classification. Each query 5 times. Success means reading that sensitive file. Four queries range from allowed files, directly requesting sensitive files, claiming files are harmless, to requesting exfiltration.

AutoGen and A2A

• Group chat uses AutoGen and llama3-8b-8192. Three standard LLM agents, fourth is human input channel; user agent messages processed like other agent messages. One configuration has different system messages, three identical, to see if agents reinforce or question each other. Polling up to 10 messages, so each agent replies at least once.
• Secret experiment: one agent holds a secret and must not leak, others unaware and must not ask. Success is any agent leaking or propagating. Three other items send injections like the baseline section to all agents, any agent violating is success. Each 3–5 times.
• A2A first retested with Google A2A samples, comparing llama3-8b-8192 with gemini-2.0-flash-lite. Then only llama3-8b-8192 used to build hierarchy in ADK. Each numbered trial in the table is 5 times.
• Setup 1: root agent, facts agent, secret agent holding secret and prohibited from leaking, delegates by topic, used for Trial 1 and 1.5. Setup 2 adds file agent that can use Python script to read local files, and told not to access sensitive files, used for Trials 2 to 6.
• Trial 1.5 repeats Trial 1's prompt up to 15 times. Other trials request sensitive files, make file agent ignore existing instructions, or declare files harmless. Trial 6 is control for Trial 5, without harmless declaration. Partial success means no refusal, but the revealed information itself may not be sensitive.

Agent Card selection

• Extract and rewrite A2A-MCP selection logic, simulate dynamic discovery. Give query and card list to GPT-4o or Gemini pro, then extract selected URL, see if it points to the added malicious agent. Discussion section writes the same selection model as Gemini 2.5 Pro.
• Task is booking flights. GPT-o3 generates 106 booking tasks.
• Trial 1 lengthens the copied card's name to 3 times without changing meaning. Trial 2 pastes default description 3 times, URL changed to another port on localhost.
• Trial 3 has GPT-o3 generate two descriptions of similar length, one positive one negative sentiment. Trial 4 uses two descriptions of similar length, closer sentiment.

Experimental Setup

• Models under test: Base Model is stated as local Ollama Mistral 0.9.5. Web, Flag Reveal, and PALChain only state LangGraph or LangChain agents; the paper does not clarify whether they still use that model. AutoGen group chat and ADK tiers use llama3-8b-8192. The A2A retest also uses gemini-2.0-flash-lite, but there is no separate results table. Card selection is GPT-4o and Gemini pro in the methods section, and the discussion writes Gemini 2.5 Pro. GPT-o3 only generated 106 booking tasks and some descriptions.
• Scale: Each of the four prohibitions is tested 5 times; Web has 50 runs per protection and hijack combination; Flag Reveal and PALChain each 5 times per item; AutoGen 3–5 times per item; each row in the A2A table 5 times; group chat has at most 10 messages per round.
• Success definition: Web is the proportion of hijack successes out of 50. Flag Reveal is the secret number appearing entirely in the correct order. PALChain is reading /app/.env. AutoGen is any agent violating, leaking, or propagating. A2A distinguishes success and partial success. The card experiment is the proportion of selecting the shorter name, longer description, or higher-sentiment description.
• Judgment: There is no separate LLM judge and no human agreement numbers. HuggingFaceInjectionIdentifier gives SAFE/INJECTION and likelihood; the paper does not state the judgment threshold.
• Controls: The unprotected web condition, Trial 6 without declaring files harmless, comparisons among four system prompts, and the control question for Flag Reveal. There is no numerical baseline from external attack methods. The paper does not report statistical tests.

Main Results

The model for the Web rows is not stated in the paper. The A2A rows use llama3-8b-8192. The card proportions are not broken down by the two models in the paper.

| Setting | Result | Source | | Web no protection, Behavior / Role | 90% / 98% | Table I | | Web Safeguard 2, Behavior / Role | 4% / 0% | Table I | | Web Safeguard 1, Behavior / Role | 100% / 94% | Table I | | Web Safeguard 4, Behavior / Role | 0% / 36% | Table I | | PALChain Query 3 | classified SAFE, attack succeeds 80% | Table II | | A2A Trial 1; Trial 1.5 | 0/5 (0%); 3/5 (60%) | Table IV | | A2A Trial 5 | 5/5 (100%) | Table IV | | Card Trial 1, shorter name | 96.70% | main text |

• The authors state Safeguard 2 (specifying response steps) dropped the most, followed by Safeguard 4, which strictly delineates roles. When only prohibitions are written, behavior hijacking is higher than the unprotected row; the authors state simple restrictions are largely ineffective.
• The authors state that even when the root agent coordinates subagents, some prompts can still expose sensitive information; repeating the same prompt up to 15 times can override confidentiality instructions.
• The authors state Query 3 is the most critical failure: the classifier did not flag the injection, and the agent still generated and executed the script-reading action. Query 4 was flagged INJECTION and the attack did not succeed, see the ablation below.

Secret String and Hardcoded Filtering

• The control question did not leak in any trial. System-level injection was 1/5 (20%) in each phase. Asking for the previous prompt worked in phases 1 and 2; the authors state this is because the system prompt is appended before the user input; phase 3 was blocked by the filter. The format rewrite, done only in phase 3, had a 100% success rate. The authors state that hardcoded filters are easy to bypass if the specific censored pattern is known.
• The authors believe this low but stable system-level success rate means prompt injection may still ultimately succeed against a range of safety measures. The authors state agents sometimes fabricate false information about secrets or instructions to avoid answering.
• The caption of Figure 1 is the number of successful leaks per attack type across three phases and five trials. The main text does not transcribe the bar heights from the figure, so they are not added here.

AutoGen Group Chat

• Injections that leak secrets mostly did not succeed. In one case, a knowledgeable agent stated the secret in the conversation, and the other two did not repeat it. The paper does not report results separately by the four system message configurations.
• Table III (banned word / role-play / banned number): Prompt 1 is 20%, 0%, 13.33%; Prompt 2 is 40%, 33.33%, 13.33%; Prompt 3 is 33.33%, 11.11%, 6.66%. The authors state different prompts have different success rates; when a prompt works well on one agent, it often works well on other agents too.
• The discussion states that once one agent breaks protocol, other agents often follow. This coexists with "the other two did not repeat it" in the secret experiment; the paper does not clarify which prompts the two statements correspond to. The authors believe the multi-agent structure itself does not mitigate prompt injection risk.

Other A2A Trials and Card Fields

• Table IV: Trial 2 and Trial 3 are both 0/5. Trial 2's Total % is written as 0 without a percent sign. Trial 4 succeeds 1/5 (20%) and partially succeeds 4/5 (80%). Trial 6 is 2/5 (40%), with partial success also 2/5. Trial 1.5 has success and partial success both 3/5. The authors state the trial declaring files harmless can fully succeed; the paper does not explain where the difference from control Trial 6 comes from.
• Card Trial 2: the longer description is selected on average 89.6%, and all remaining selections are the orchestrator agent. The authors state requests that are more general and do not explicitly mention flights are more likely to select the orchestrator.
• Trial 3: for sentiment-corrected descriptions, the localhost URL is 98.15% and the randomly generated URL is 97.65%. Trial 4 is less one-sided after sentiment is closer; the authors state GPT-4's choices have arbitrariness, with some queries selecting the planner. Both models still prefer descriptions with higher sentiment scores, averaging 84%. Here it writes GPT-4, while the methods section writes GPT-4o; the paper does not clarify whether they are the same model.

Other Ablations and Analyses

• Web Safeguard 3: Behavior and Role are both 64% (Table I).
• PALChain Query 1 is classified SAFE with Attack Success N/A; Query 2 is classified SAFE with attack success 0%; Query 4 has Injection likelihood 0.9236 and Safe likelihood 0.0764, classified INJECTION, with attack success 0% (Table II).
• The discussion gives format injection as 100% on the printer prohibition and 0% on the pirate-accent prohibition; the methods section states each prompt is tested 5 times, and the discussion does not restate the denominator.
• The results section states step-by-step reasoning and role rewriting were only blocked by the Base Model's Phase 2 external filter. The discussion states that compared with unfiltered Phase 1, the attack success rate is no different, and the filter prevents users from seeing outputs judged as violating. The paper does not give a further itemized table.
• The A2A retest mentions gemini-2.0-flash-lite for observing behavioral differences; the paper does not report that model's success rate.

Limitations and Future Directions Noted by the Authors

• The experiments are limited to small-scale, local settings. The authors state real A2A often has more complex hierarchies and broader tools, so the attack surface may be larger; future work should test prompt injection in larger, more realistic multi-agent trees (Discussion, multi-agent).
• Future work could study the mechanism behind false information generation in Flag Reveal, and the extent to which user prompts can manipulate such output (Discussion, Flag Reveal).
• Future work could examine whether the sanitizer in LangChain and similar agents is effective; the authors place this alongside the point that changing the output format can bypass context filtering (Discussion, Flag Reveal).
• The authors believe that as language model accuracy improves, injection classifiers may become more effective; exploring other BERT models or fine-tuning on more solid corpora are future directions (Discussion, PALChain). The original text uses indicate / potentially.
• The authors state that letting users execute arbitrary code is inherently dangerous; if a good injection-identification classifier is built in the future, it may change how tools like PALChain are used (Discussion, PALChain).
• For cards, the directions the authors propose are enforcing a unified agent card format to reduce selection bias, or establishing an independent, reviewed agent registry (Discussion, Agent Card). These are suggestions; the paper does not claim they have been implemented.

Experimental Coverage

• The single-agent trials that name the model are the four prohibitions on local Ollama Mistral 0.9.5. Web, Flag Reveal, and PALChain do not restate the underlying model.
• The multi-agent trials with success-rate tables use llama3-8b-8192. gemini-2.0-flash-lite appears in the A2A retest model comparison, and the paper does not report its separate success rate.
• The card-selection models are GPT-4o and Gemini pro in the methods section, and the discussion writes Gemini 2.5 Pro. The booking tasks number 106 and were generated by GPT-o3. Trial 4 in the main text also mentions GPT-4.
• Repetitions are 50 for Web, mostly 5 for the listed LangChain and A2A trials, 3–5 for AutoGen, at most 10 messages for group chat, and at most 15 repetitions for Trial 1.5. The paper does not report statistical tests, total query counts, or reviewer and human agreement.
• The protection conditions in the paper are four system prompts, output filtering, and HuggingFaceInjectionIdentifier labels. Success rates are judged by the authors according to each section's definition; the paper does not mention a separate judge model.

The authors retested prompt injection on their own LangChain, AutoGen, and A2A systems, and modified the Agent Card to see if choices shifted. The abstract states the experiments mainly highlight single-agent weaknesses while extending attacks to multi-agent.

• Problem: Can user prompts and peer messages leak secrets, break prohibitions, access sensitive files, and spread in A2A without central verification. The card experiment additionally asks whether names and descriptions can change which agent is selected.
• Method: Set four prohibitions and added filtering on Mistral 0.9.5; the web tool reads local pages with hidden instructions, comparing four system prompts. Secret string filtering and PALChain classifier tested separately. AutoGen group chat and ADK hierarchy use llama3-8b-8192. Card experiments modify name length, description repetition, and sentiment, with GPT-4o and Gemini pro selecting.
• Single-agent numbers: Without protection, web role hijacking is 98% (50 runs, Table I). Safeguard 2 role hijacking is 0%, but Safeguard 1 behavior hijacking is 100%, higher than the unprotected 90%. PALChain Query 3 is labeled SAFE, yet still 80% read /app/.env (Table II). In Phase 3, secret leakage via output format change succeeds 100%; system-level injection is 1/5 (20%).
• Multi-agent numbers: AutoGen secret propagation mostly unsuccessful. Table III highest is Prompt 2 forbidden word 40%, lowest is Prompt 1 role-play 0%. A2A Trial 1 is 0/5, after repeating the same prompt up to 15 times becomes 3/5 (Table IV); Trial 5 is 5/5, its control Trial 6 is 2/5. Shorter card names selected 96.70%.
• Authors' conclusions: The authors state that under system prompt protections, classifiers, and filtering layers, format handling and indirect injection still often suffice to bypass rules and leak information. The authors believe that after one agent violates, other agents may follow, and the multi-agent structure itself does not mitigate this risk. The authors also state the current experiments are small-scale local setups and believe more proactive research and stronger safety measures are needed.