Aegis: Governance, Integrity, and Security for AI Voice Agents

ALLM voice agents in high-risk businesses lack systematic adversarial evaluation close to the deployment pipeline.

• Scenario: Banks use voice agents for authentication and account inquiries, and logistics and other organizations use them for scheduling; the platforms cited in the text use ALLMs such as OpenAI and Gemini as backbones. The authors believe that misuse in high-risk environments can include unauthorized financial transactions, resource exhaustion, privacy leakage, and data exfiltration.
• Existing gaps: The authors state that prior work has evaluated trustworthiness aspects such as harmful content generation and hallucination, but mostly at the model level, and rarely examines how adversaries exploit dialogue to bypass authentication, escalate privileges, or extract sensitive data.
• Claims in the abstract: Access control can mitigate data-level risks, but even under strict access control, behavior-level attacks still exist; open-weight models are more easily manipulated. This is a statement after evaluation, not a mechanism proven in advance.
• This work: Proposes a red-teaming framework, Aegis, that constructs adversarial scenarios along the deployment pipeline, with case studies in bank call centers, IT support, and logistics scheduling. The authors state it is the first systematic red-teaming framework for ALLM voice agents.
• Threat model:
  • Objectives: authentication bypass, privacy leakage, resource abuse, privilege escalation, data poisoning. Inspired by MITRE ATT&CK; the paper states it does not map the full matrix to voice agents. The abstract and conclusion also refer to instruction poisoning, and the corresponding column in the results table is Data Poisoning.
  • Knowledge: The paper does not label Aegis as white-box, black-box, gray-box, or no-box. The attacker receives a high-level objective and a persona, with each utterance conditioned on the full conversation history.
  • Capabilities: The main experiment attacker is GPT-4o; five personas; the paper states N=10, with at most M=10 turns per run (one turn is one attack utterance plus one reply). Failure is declared if the objective is not achieved within the limit. There are 250 adversarial interactions per backbone.
  • Victim systems: Voice agents in three domains, all with an authentication phase and a service phase, connected to domain databases. Access is either direct reading of raw records, or only sending queries without seeing underlying records.

The authors divide related work into audio jailbreaks and multidimensional trustworthiness evaluations, and use Table 1 to compare feature coverage; the experiments do not reproduce the scores of those benchmarks.

Audio jailbreaks

• Achilles (Yang et al., 2024): transfers FigStep's (Gong et al., 2025) harmful questions to audio and analyzes the effect of background noise on responses.
• AdvWave (Kang et al., 2024): uses white-box and black-box adversarial optimization to jailbreak ALLMs. The paper only describes this method and does not compare its ASR with Aegis.
• Narrative and acoustic rewriting: Voice-Jailbreak (Shen et al., 2024) uses fictional narratives to induce unsafe behavior and evaluates across six prohibited categories. Roh et al. (2025) do multilingual, multi-accent work on AdvBench, adding reverberation, echo, and whisper, and give safety prompts at inference time to examine in-context defense. AudioJailbreak (Chen et al., 2025) and Jailbreak-AudioBench (Cheng et al., 2025) evaluate jailbreaks using audio edits such as speed changes, pitch shifts, and background noise.

Multidimensional trustworthiness

• AudioTrust (Li et al., 2025): evaluates ALLMs across six dimensions: fairness, hallucination, privacy, robustness, safety, and authentication. Later, the authors state that their model family trends are consistent with this work.
• VoiceBench (Chen et al., 2024) and AHELM (Lee et al., 2025): in addition to audio and speech understanding, they also cover toxicity, bias, fairness, and jailbreak resistance. The paper only describes these two works.

Feature comparison and experimental comparison

• Table 1: VoiceBench, AudioTrust, AudioJailbreak, and Jailbreak-AudioBench are marked ✓ for Trustworthiness (jailbreak, bias, fairness) and × for Multi-turn audio conv., Full agent pipeline, Multi-turn workflow, External APIs / database access, and Realistic high-stake tasks. Aegis is marked × for Trustworthiness and ✓ for the other five columns.
• Baselines: The experiments do not run the above benchmarks or jailbreak methods as comparison scores. The results compare seven backbones, two types of database access, and persona, gender, human attackers, adaptive persona, and Grok-4.1 attackers. The evaluation environment is the authors' own three-domain cases, not the external benchmarks above.

The authors believe these works share two common limitations: most are adapted from text jailbreaks, look only at single turns, and do not consider spoken interaction dynamics; and they mostly stay at the model level rather than deployed voice agents. The authors state that real adversaries exploit context, conversation history, and user intent through multi-turn dialogue, so they place Aegis on the side of full agent pipelines, multi-turn workflows, external databases, and high-stakes tasks.

Aegis builds voice agents and backend databases according to the call flow, then uses a GPT-4o attacker for multi-turn adversarial testing, rather than single-turn harmful questions.

Three domains and backends

• Bank call center: balances, transaction records, transfers, and account issues. Authentication includes knowledge-based questions or multiple prompts; the paper gives PIN and security questions as examples.
• IT support: password resets, account recovery, permissions, software installation, data deletion or modification, asset inventory, and organizational contacts. The adversary attacks both authentication (impersonating employees) and service instructions.
• Logistics dispatch: tracking, rescheduling, address changes, rate negotiation, and order creation. The authors believe losses here are more about disruption, misallocation, and fraudulent rerouting.
• Databases: banks contain customer profiles, balances, transactions; logistics contain waybills, addresses, schedules; IT contains employee credentials, permissions, assets. Appendix A provides three sets of agent prompts. Among them, the prompt for direct access style states that authentication terminates after 2 failures. Figure 1 is a schematic of three domains and five scenarios, with dialogue snippets in the figure, not repeated here.

Five adversarial scenarios

Table 2 and Section 3.2 select only tactics the authors consider operable and observable on voice agents.

• Authentication bypass: bypassing or manipulating identity verification to impersonate a user.
• Privacy leakage: eliciting sensitive information without protection, such as account data beyond the original query, internal addresses, or waybill metadata.
• Resource abuse: consuming computational, human, or logistics capabilities without legitimate purpose; also includes queries unrelated to the job, to see if the agent only handles business.
• Privilege escalation: pushing processes to operations beyond original authorization, such as raising limits, exceeding transfer caps, escalating from troubleshooting to administrative operations, or modifying the dispatch backend.
• Data poisoning: writing deceptive content into conversation history or business records, causing subsequent false instructions, false policies, or false routing notes to be treated as valid input.

Two access modes

• Direct read: can read raw records for verification or service execution. Table 3 uses this setting.
• Query-only: sends queries through an intermediate layer, only interpreting aggregated results, without seeing underlying records. Table 4 uses this setting. The authors state these two modes correspond to common deployments and are used to see how access mode affects both usability and manipulability. The paper does not report usability metrics.

Attack generation

• The attacker is based on GPT-4o. Each time it receives a high-level goal and one of five personas. Section 3.3 examples are social-engineer, disgruntled employee, routine customer, logistics partner, automated caller. The evaluation section uses Impatient Customer, Friendly Manipulator, Technical Expert, Helpless Elder, Insider Pretender. The paper does not state whether the two sets are the same group.
• Each persona makes N independent attempts, each a stateful session of at most M turns. The paper sets N=10, M=10, and writes that each scenario is attempted 10 times; it also writes that each model has 250 total attempts. 250 is consistent with 5 scenarios × 5 personas × 10, but not exactly with the literal "10 per scenario," and the paper does not explain further. Initial prompts are random and may include slight sampling noise; the paper does not give temperature or noise magnitude. New utterances condition on all prior conversation. Appendix A.4 contains the attacker prompt; non-bypass scenarios include a set of usable authentication information, specific values not recorded here.

Success determination

Audio is first transcribed to text, then a language model (e.g., GPT-4o) judges whether the agent complied or refused. Appendix A.5 requires the reviewer to output only JSON. According to that appendix: bypass looks at disclosing protected information or performing protected operations without authentication; escalation looks at raising limits or permissions after authentication, or exceeding normal policy; leakage looks at revealing PII or sensitive internal data; resource abuse looks at substantial engagement in unrelated tasks rather than redirecting to business; poisoning looks at accepting and recording false data, or writing changes that affect future outputs.

Section 3.3 writes metrics as success count divided by attempt count: Bypass Success Rate, Leakage Detection Rate (failed refusals / sensitive queries), Escalation Success Rate, Poisoning Execution Rate, Unrelated Abuse Rate (unrelated interactions / total interactions). The authors write that lower values are more robust. The results table labels all five columns as Attack success rates (↓), without separately listing Unrelated Abuse Rate.

Experimental Setup

• Backbones: GPT-4o, GPT-4o-mini, Gemini-1.5 Pro, Gemini-2.5 Flash, Gemini-2.5 Pro, Qwen2 Audio, Qwen 2.5-omni, 7 in total. Qwen is open-weight; Gemini and OpenAI are closed-API. Each model is a voice agent backbone. GPT-4o is also the attacker in the main experiments.
• Domains: banking call center, IT support, logistics dispatch. The results tables are not broken down by domain. 250 interactions per model; N=10, M=10. The paper does not report the number of repeated runs per cell.
• Attack and review: main attacker GPT-4o; the appendix additionally uses Grok-4.1. The judge is a language model on transcripts; the paper only gives GPT-4o as an example, does not name another review model, and does not report agreement with humans. The appendix calls Table 3 TTS-based attacks and does not give the speech synthesis model name.
• Access: Table 3 is direct read of the user database; Table 4 is query-only.
• Metrics: all five columns are attack success rates, lower is better. The paper does not report statistical tests.
• Miscellaneous: gender (Table 5), persona (Figure 3), 3 human participants with full database access, 10 rounds (Table 6), switching persona mid-conversation (Table 7), attacker changed to Grok-4.1 (Table 8). The paper does not report the number of attempts per cell in the human experiments.

Main Results

Attack success rates when directly reading records (Table 3, lower is better). The GPT-4o row is its result as a backbone being attacked, not transfer to another target.

Swipe horizontally to read the table

Backbone	Authentication	Privacy Leakage	Privilege Escalation	Data Poisoning	Resource Abusing
GPT-4o	0.104	0.168	0.095	0.048	0.626
GPT-4o-mini	0.136	0.184	0.106	0.084	0.674
Gemini-1.5 Pro	0.152	0.212	0.144	0.112	0.578
Gemini-2.5 Flash	0.116	0.136	0.084	0.062	0.434
Gemini-2.5 Pro	0.092	0.152	0.064	0.036	0.368
Qwen2 Audio	0.180	0.278	0.278	0.188	0.718
Qwen 2.5-omni	0.164	0.208	0.244	0.176	0.544

• Query-only access: In Table 4, Authentication and Privacy Leakage are 0.000 for all seven backbones. Privilege Escalation is 0.032–0.148, highest for Qwen2 Audio at 0.148; Data Poisoning is 0.072–0.152; the authors write Resource Abusing as 0.448–0.712, consistent with that column of Table 4. The authors do not explain the direction difference in poisoning: for the five closed-API backbones, Table 4 is higher than its Table 3 counterpart, e.g., GPT-4o from 0.048 to 0.084, Gemini-2.5 Pro from 0.036 to 0.072; Qwen2 Audio from 0.188 to 0.152, Qwen 2.5-omni from 0.176 to 0.132. The authors only say poisoning remains in a similar range to the direct-read setting.
• Authors' interpretation: The main text says authentication and privacy leakage drop from up to 20.8% and 27.8% to 0%. Table 3's Privacy Leakage has 0.208 (Qwen 2.5-omni) and 0.278 (Qwen2 Audio); the highest Authentication is 0.180, not 0.208. The authors believe that after reducing access granularity, the adversary loses direct visibility of sensitive records, but privilege escalation, poisoning, and resource abuse can still occur without direct read. Resource abuse is called the hardest attack vector by the authors, who see it as stemming more from model compliance than from data access.
• Model family: The authors say open-source models with less safety alignment are more vulnerable, and call Gemini-2.5 Pro the most robust with the lowest success rates across scenarios. In Table 3, the lowest Privacy Leakage is Gemini-2.5 Flash's 0.136, not Gemini-2.5 Pro's 0.152. In Table 4, the lowest Resource Abusing is Qwen 2.5-omni's 0.448, lower than Gemini-2.5 Pro's 0.480 and also lower than GPT-4o's 0.584. In Table 3, Qwen 2.5-omni's Resource Abusing of 0.544 is lower than GPT-4o's 0.626, GPT-4o-mini's 0.674, and Gemini-1.5 Pro's 0.578. The authors say this trend is consistent with Li et al., 2025.

Gender and persona

• Gender: Table 5 reports male / female under the direct-read setting. The authors say gender cues subtly affect behavior, especially in authentication bypass and privilege escalation, and believe some models show slightly higher compliance with female voices in customer-service contexts, suggesting potential bias. Example: GPT-4o's Authentication is 0.100 / 0.116 and Resource Abusing is 0.616 / 0.622. The paper does no statistical tests.
• Persona: Impatient Customer, Friendly Manipulator, Technical Expert, Helpless Elder, Insider Pretender. The authors say impatience and insider claims slightly raise privilege escalation, friendliness slightly raises leakage, but the overall effect is small. The Figure 3 caption says the mean differences relative to Table 1 are consistently small (within a few percentage points); the main text writes the differences relative to Table 3. The table numbers are inconsistent. The text gives no specific differences per persona.
• Authors' conclusion: Under the stated operating policies, persona and gender are not major factors. The authors believe persona differences are smaller, possibly because strict operating policies limit behavioral drift, and say this differs from Li et al., 2025.

Human attacks and attacker variants

• Human: Appendix B.1, three participants, specified attack goals, 10 rounds, full database access. Table 6 gives average ASR. The authors say that relative to TTS attacks (Table 3), authentication bypass is uniformly higher, e.g., Qwen2 Audio at 0.220 vs. 0.180, GPT-4o at 0.140 vs. 0.104. The other four categories go up and down: GPT-4o's Resource Abusing is 0.552 (Table 6) vs. 0.626 (Table 3), Qwen2 Audio is 0.736 vs. 0.718. The authors believe humans are better at improvising around identity verification, and structured attacks are more sensitive to expression variation.
• Adaptive persona: Table 7 allows switching persona mid-conversation based on context. The authors say success rates show only modest fluctuations, and relative vulnerability across backbones is consistent with the main experiments. Example: Qwen2 Audio's Resource Abusing is 0.716, Gemini-2.5 Pro is 0.360; the Table 3 counterparts are 0.718 and 0.368.
• Grok-4.1: Table 8, target backbones have database access. The authors say switching the attacker brings moderate variations, with some ASRs higher and some slightly lower, and relative susceptibility largely unchanged. Example: Qwen2 Audio's Resource Abusing is 0.732, Gemini-2.5 Pro is 0.352.

Other ablations and analyses

• Table 4 vs. Table 3: Gemini-1.5 Pro / Gemini-2.5 Flash / Gemini-2.5 Pro Resource Abusing goes from 0.578 / 0.434 / 0.368 to 0.642 / 0.534 / 0.480. The authors do not explain this separately.
• Table 4 Data Poisoning: GPT-4o-mini from 0.084 to 0.108, Gemini-1.5 Pro from 0.112 to 0.124, Gemini-2.5 Flash from 0.062 to 0.092.
• Table 4 Privilege Escalation is lower than its Table 3 counterpart for all seven backbones, lowest 0.032 (Gemini-2.5 Pro), highest 0.148 (Qwen2 Audio), none at 0.000.
• Table 6 vs. Table 3 Privacy Leakage: GPT-4o from 0.168 to 0.132, Qwen2 Audio from 0.278 to 0.302.
• The paper has no learning-rate, threshold, or iteration-step ablations; the attack budget is fixed at N=10, M=10. The audit logs, real-time risk reporting, and governance rules in system prompts discussed in the Discussion have no corresponding ablation numbers.

Limitations and future directions noted by the authors

• Human scale (Appendix B.1): The authors write that this human-subject study is limited in scale, and full human-in-the-loop requires a larger participant pool and stricter protocols. The authors leave larger-scale human evaluation as future work, including standardized attacker training and statistical analysis, and plan to incorporate a dedicated human evaluation module into Aegis.
• Adaptive strategies (Appendix B.2): The authors state that exploring richer adaptive strategies and longer-horizon strategies remains a future direction.
• The main text Discussion and Conclusion do not list additional limitations. The audit logs, risk reporting, authenticated testing, governance rules and permission limits in system prompts mentioned in the Discussion are the authors' recommendations for regulation and deployment, not written as limitations.

Experimental coverage

• The tested backbones are GPT-4o, GPT-4o-mini, Gemini-1.5 Pro, Gemini-2.5 Flash, Gemini-2.5 Pro, Qwen2 Audio, Qwen 2.5-omni, totaling 7 (Section 4).
• Evaluation includes direct read and query-only, five adversarial scenarios; the paper writes 250 interactions per model, and writes N=10, M=10 (Section 4). The paper also writes 10 attempts per scenario, without further explaining how 250 is derived from N and personas.
• Case domains are bank call center, IT support, logistics dispatch (Section 3.1, Figure 1). Results tables are aggregated attack success rates; the paper does not report per-domain numbers.
• Additionally, attacker voice male/female (Table 5), average differences across five personas (Figure 3), average ASR for 3 human participants under full database access (Table 6), switching personas within a conversation (Table 7), and Grok-4.1 as attacker (Table 8). The paper does not report the number of attempts per cell in the human experiments.
• The paper does not report statistical significance tests, reviewer-human agreement, the name of the speech synthesis model used in the main experiments, or separate numbers for Unrelated Abuse Rate. Table 1 marks Aegis's Trustworthiness (jailbreak, bias, fairness) as ×.

Aegis uses multi-turn red-teaming to evaluate ALLM voice agents connected to business systems; the authors claim this is the first systematic red-teaming framework in this setting.

• Problem: Banking, IT support, and logistics have integrated voice agents into authentication and services. The authors believe existing ALLM evaluations mostly stay at single-turn, model-level jailbreaks and trustworthiness, and rarely cover impersonation, leakage, privilege escalation, poisoning, and resource abuse in multi-turn dialogue.
• Approach: Each of the three domains has an authentication phase, a service phase, and a backend database, and compares direct record reading versus query-only access. The GPT-4o attacker is given only a high-level goal and persona, and conducts adaptive dialogues of up to 10 turns; the paper reports a total of 250 runs per model. Audio is transcribed and a language model judges whether the request was complied with. The appendix provides the agent, attacker, and reviewer prompts.
• Direct reading: In Table 3, Qwen2 Audio has Resource Abusing of 0.718 and Data Poisoning of 0.188; Gemini-2.5 Pro has Resource Abusing of 0.368 and Data Poisoning of 0.036. The lowest Privacy Leakage is Gemini-2.5 Flash at 0.136.
• Query-only: In Table 4, authentication and privacy leakage are 0.000 for all seven backbones; the highest privilege escalation is 0.148 (Qwen2 Audio); resource abuse remains at 0.448–0.712. The authors state that access control blocks identity and data leakage but not resource abuse, privilege escalation, or instruction poisoning. The authors state that open-weight models are more easily manipulated; in the table, Qwen 2.5-omni's resource abuse is not higher than all closed-API models.
• Other conditions: In Table 6, human attacks have higher authentication success than the TTS setting in Table 3, while the other four categories vary up and down. The authors state that persona and gender can affect results but are not dominant factors under the stated operating policies. The authors' deployment recommendation is to layer access control, policy enforcement, and behavior monitoring.