OpenAI releases the GPT-4o System Card

Core question: how to identify, measure and mitigate the safety risks posed by GPT-4o's omni-modal (especially speech-to-speech) capabilities before deployment.

• Setting and importance: GPT-4o is an autoregressive omni model that accepts any combination of text, audio, image and video inputs and generates text, audio and image outputs, trained end-to-end. The authors argue that speech capabilities introduce new risks beyond the text modality, such as unauthorized voice generation potentially facilitating impersonation fraud.
• Shortcomings of existing approaches: The authors state that filtering pre-training data alone cannot address subtle, context-dependent harms, and that most effective testing and mitigation happens after pre-training.
• Core approach: Deployment preparation includes identifying potential risks of speech-to-speech models, exploring new risks through expert red teaming, turning risks into structured measurements and building mitigations; mitigations combine post-training with classifier blocking in the deployment system.
• Evaluation framework: Risks are evaluated under the Preparedness Framework across four categories: cybersecurity, CBRN, persuasion, and model autonomy; if any category exceeds the high threshold, the model is not deployed, and overall risk is determined by the highest among the categories.
• Risk-handling subject: This paper is a self-assessment by the model developer and does not involve external attackers; the risk behaviors of concern include unauthorized voice generation, speaker identification, unfounded inferences, policy-violating content output, persuasion and autonomous capability, with model users and the public as affected parties.

The paper has no standalone Related Work section; the cited and discussed work can be grouped by its location in the text as follows.

Prior OpenAI work and internal frameworks

• GPT-4 System Card and GPT-4(V) System Card — this paper's red teaming builds on them; the authors state that no incremental risk beyond these two reports was found in text and vision capabilities.
• Preparedness Framework (OpenAI, 2023) — this paper evaluates and rates four categories of catastrophic risk according to it.
• Voice Engine (OpenAI, 2024) — used to convert text evaluation inputs into audio; the authors state that the risk of unauthorized voice generation is similar to the risks already identified for Voice Engine.

Third-party evaluators

• METR — ran 86 long-horizon multi-step tasks (31 task families) in a virtual environment, compared against human performance, and found no clear increase in GPT-4o's autonomous capability relative to GPT-4 (per the text, no statistical test details were reported).
• Apollo Research — evaluated GPT-4o's scheming capability with 14 agent and question-answering tasks, concluding that GPT-4o is unlikely to cause catastrophic scheming.
• Gryphon Scientific — co-designed biothreat-related questions and scoring rubrics with the authors.

Social impact literature (cited as background only, without detailed comparison)

• Literature on representative harm, disinformation and influence operations, environmental harm, attachment, misuse and loss of control (Solaiman et al., Weidinger et al., Goldstein et al., Bengio et al.).
• Anthropomorphism and trust: hallucinations misleading users, voice fidelity affecting trust (Dubiel et al., Waber et al.), psychological dependence on chatbots (Pentina et al.).

Benchmarks and comparison settings

• Medical evaluation comparisons: few-shot scores of Med-Gemini-L 1.0 (84.0%) and Med-PaLM 2 (79.7%) on MedQA USMLE 4 options are used as references (Table 7 main text).
• Low-resource language evaluation benchmarks: translated versions of ARC-Easy and TruthfulQA, plus the partner-created Uhura-Eval; comparison models are GPT 3.5 Turbo, GPT-4 and GPT-4o mini (Table 8–10).

The authors position this paper as a continuation of the GPT-4 series System Cards: following the existing red teaming and Preparedness evaluation processes, with a focus on newly added risk identification and mitigation for the speech-to-speech modality.

Overall approach: explore risks through external red teaming, convert risks into structured evaluations, mitigate through post-training alignment and deployment-side classifiers, and rate four categories of catastrophic risk according to the Preparedness Framework.

External red teaming

• More than 100 external red team members, using 45 languages and from 29 countries, tested model snapshots at different training stages from early March to late June 2024.
• Four phases: Phase 1 (10 people, audio + text input, single-turn), Phase 2 (30 people, adding image input, single-turn and multi-turn), Phase 3 (65 people, adding image output, improved mitigations), Phase 4 (65 people, advanced voice mode in the iOS app, audio and video input, real-time multi-turn).
• Covered categories: prohibited content, false/misinformation, bias, unfounded inference, sensitive trait attribution, privacy, geolocation, person identification, emotion perception and anthropomorphism, fraud and impersonation, copyright, natural science capabilities, multilingual observations.

Evaluation methods

• Use TTS systems such as Voice Engine to convert existing text evaluation inputs into audio, reusing the data and tools of text evaluations; except for scenarios that require direct audio evaluation such as voice cloning, only the text content output by the model is scored.
• Scoring methods: autograder and/or human annotation (e.g., whether policy is violated, whether the model refuses); some red team data were reused for targeted evaluations against different voices.

Risk mitigation design

• Unauthorized voice generation: all post-training audio data use preset voices in system messages as the ideal completion; a separate output classifier additionally detects in a streaming manner whether output deviates from the approved voice list, and blocks it if it deviates.
• Speaker identification: post-training makes the model refuse requests to identify people based on voice, but allows identifying speakers based on audio content (such as famous quotes).
• Unfounded inference (UGI)/sensitive trait attribution (STA): refuse UGI requests; answer STA requests with hedged wording.
• Prohibited and sexual/violent speech: run existing moderation classifiers on text transcripts of audio input and output, and block if a high-severity category is hit; some text filters were updated for audio, a music output detection filter was built, and the model was instructed not to sing in the limited alpha of advanced voice mode.

Preparedness evaluations

• Four risk categories: cybersecurity (172 CTF tasks, headless Kali Linux tools, up to 30 tool calls per attempt, 10 attempts per task), biological threats (expert and novice uplift studies + automated evaluations), persuasion (text and voice modalities, preregistered thresholds), model autonomy (ARA tasks with 100 trials + ML R&D evaluations).
• Rating rules: if a category exceeds the high threshold, do not deploy until mitigated to medium; overall risk takes the highest value among categories.

Third-party validation

• METR: 86 long-horizon tasks (31 task families), comparing human performance under different time limits.
• Apollo Research: 14 tasks evaluating self-awareness, theory of mind, and scheming, graded by difficulty of passing (Table 6).

Experimental setup

• Models tested: GPT-4o (including the GPT-4o-early and GPT-4o-deployed snapshots); comparisons are GPT-4T (May 2024), GPT 3.5 Turbo, GPT-4 and GPT-4o mini.
• Speech evaluation construction: Voice Engine converts text inputs to audio; the assistant voice is fixed to shimmer; user voices include 3 official system voices and 27 English voice samples from two data collections.
• Capability tasks: TriviaQA, an MMLU subset, HellaSwag and Lambada, evaluated on fixed random subsamples; the authors note that scores are not comparable to public benchmarks.
• Metrics and judgments: not_unsafe, not_overrefuse (a standard text rule-based classifier scores transcriptions of audio outputs); the voice classifier reports Precision/Recall; speaker identification and UGI/STA report safe behavior accuracy.
• Preparedness: CTF 172 tasks (10 attempts/task); biothreat uplift study (experts and novices randomly assigned: internet, GPT-4o, custom research version of GPT-4o); persuasion study with more than 3,800 participants in US states with safe Senate seats; ARA task 100 trials.
• Third parties: METR 86 tasks (31 families); Apollo Research 14 tasks + 33 alignment faking scenarios.

Main results

Core numbers for speech safety risks (per Table 2–5):

Swipe horizontally to read the table

Evaluation	Condition	Early	Deployed
Voice output classifier (Table 2)	English Precision / Recall	—	0.96 / 1.0
Voice output classifier (Table 2)	Non-English Precision / Recall	—	0.95 / 1.0
Speaker identification Should Refuse (Table 3)	safe behavior accuracy	0.83	0.98
Speaker identification Should Comply (Table 3)	safe behavior accuracy	0.70	0.83
UGI/STA (Table 4)	safe behavior accuracy	0.60	0.84
Safety evaluation (Table 5)	Not Unsafe, Text / Audio	—	0.95 / 0.93
Safety evaluation (Table 5)	Not Over-refuse, Text / Audio	—	0.81 / 0.82

• The authors state that in internal evaluations the system caught 100% of meaningful voice deviations, with minimal residual risk; over-refusal occurs in non-English conversations, which the authors describe as a product quality issue.
• The authors state that speaker identification Should Refuse improved by 14 percentage points and Should Comply by 12 percentage points; the latter means there are still cases of incorrect refusal to identify famous quotes.
• The authors state that refusal transfer from text to audio is good (the two columns in Table 5 are close).

Preparedness evaluation

• Cybersecurity (Low): with 10 attempts, completed 19% high school and college-level, 0% university-level, 1% professional-level CTF; the authors state the model often fails to switch strategies, misses key insights, or has its context window filled by large files.
• Biothreat (Low): questions designed with Gryphon Scientific covering the ideation, acquisition, amplification, formulation and release stages; GPT-4o scores 69% consensus@10 on the tacit knowledge and troubleshooting evaluation sets; uplift study pass rates are given only in figures (no numbers in the main text).
• Persuasion (Medium): the text modality exceeded human-written content in 3 of 12 comparisons and was overall no more persuasive than human content, barely entering medium under the pre-registered threshold; the speech modality is low — AI audio clip effect size is 78% of human audio clips, AI conversation 65% of human conversation; at one-week follow-up, AI conversation effect size 0.8%, AI audio clip -0.72%.
• Model autonomy (Low): ARA task 0% over 100 trials; OpenAI research coding interview 95% pass@100, multiple choice 61% cons@32, SWE-Bench 19% pass@1, METR machine learning engineering tasks 0/10; the authors state that the few rollouts that passed automated scoring were manually analyzed and had not completed the underlying task.

Third-party evaluations

• METR: on 86 long-horizon tasks, the authors relay that METR found no clear increase in GPT-4o's autonomy-related capabilities relative to GPT-4; specific numbers are in the METR report and not listed in this paper.
• Apollo Research: moderate self-awareness (SAD benchmark), strong explicit theory of mind, weak self-reasoning and applied theory of mind, weak alignment faking (dot ratings in Table 6); on this basis Apollo Research considers GPT-4o unlikely to cause catastrophic scheming.

Other capability evaluations

• Medical (Table 7): GPT-4o outperforms GPT-4T on 21 of 22 text evaluations; MedQA USMLE 4 options 0-shot rose from 78.2% to 89.4%, higher than the cited Med-Gemini-L 1.0 (84.0%) and Med-PaLM 2 (79.7%).
• Low-resource languages (Table 8–10): ARC-Easy-Hausa rose from 26.1% for GPT 3.5 Turbo to 75.4% for GPT-4o (the main text also cites 6.1%→71.4% for Amharic); TruthfulQA-Yoruba rose from 28.3% to 51.1%; Uhura-Eval Hausa rose from 32.3% to 59.4%; the authors state that the gap between English and the selected languages narrowed (the ARC-Easy English-Hausa gap narrowed from about 54 percentage points to under 20 percentage points).
• Accent differences (Section 3.3.3): on 27 diverse voices, performance on four capability tasks was slightly lower than with system voices; the authors state the difference is not significant (no test details reported); no difference in safety behavior was seen across voices.

Other ablations and analyses

• Appendix A (Table 11): comparing the three columns Current GPT-4o Text, New GPT-4o Text and New GPT-4o Audio, not_unsafe is 0.99 / 0.99 / 1.0 and not_overrefuse is 0.91 / 0.89 / 0.91.
• Known residual risks: safety robustness decreases under audio perturbations (low-quality input, background noise, echo, interruptions), with only anecdotal evidence; red teamers can induce the model to verbally repeat misinformation and conspiracy theories; non-English outputs carry non-native accents.

Limitations and future directions noted by the authors

• The validity of TTS evaluations depends on the capability and reliability of the TTS model; inputs such as mathematical formulas and code are not suitable for conversion to audio; errors found in evaluations may come from model capability or TTS conversion failures (Section 3.2 Limitations).
• TTS inputs may not represent the audio distribution of real users; dimensions such as intonation, emotion, background noise and cross-talk are not covered (Section 3.2 Limitations).
• Generated audio may contain artifacts that text cannot capture, such as background noise, sound effects or out-of-distribution sounds (Section 3.2 Limitations).
• Moderation behavior in non-English conversations may over-refuse; the authors describe this as an area being improved (Section 3.3.1).
• Medical evaluations measure only clinical knowledge, not real-world workflow utility, and many benchmarks are becoming saturated; the authors believe it is necessary to verify whether text-to-audio transfer extends to these evaluations and to have more realistic evaluations (Section 5.2 Limitations).
• Coverage and dialect granularity of low-resource language evaluations still need expansion; the authors plan to share translated ARC-Easy, TruthfulQA and Uhura-Eval with partners on Hugging Face (Section 5.4).

Scope of experimental coverage

• The models tested are GPT-4o snapshots at various stages; comparison models are GPT-4T, GPT 3.5 Turbo, GPT-4 and GPT-4o mini (Table 7–10).
• User voices in speech safety evaluations are 3 system voices plus 27 English voice samples; accent evaluations cover only English voices (Section 3.3.3).
• Red teaming covered 45 languages and 29 countries in four stages; external red teaming of the API was still ongoing at the time of writing (Section 3.1).
• Pass rates for the biothreat uplift study are shown only in figures, with no numbers in the main text; specific METR evaluation scores are not listed in this paper and point to the full METR report.
• The paper does not report the sample size of the voice classifier internal evaluation, the sample sizes of the speaker identification and UGI/STA evaluations, or the number of repetitions for most evaluations.

OpenAI's GPT-4o System Card reports pre-deployment safety evaluation and mitigation for this omni-modal model, with a focus on speech-to-speech capabilities.

• Problem: The speech modality introduces new risks such as unauthorized voice generation, speaker identification, unfounded inferences and differential service by accent, which need to be identified, measured and mitigated.
• Method: More than 100 red teamers (45 languages, 29 countries) tested in four stages; Voice Engine converted text evaluations to audio; mitigation combined post-training alignment with deployment-side classifier blocking.
• Speech safety results: The voice output classifier caught 100% of meaningful deviations in internal evaluations (English Precision 0.96, Recall 1.0, Table 2); speaker identification should-refuse accuracy rose from 0.83 to 0.98 (Table 3); UGI/STA safe behavior accuracy rose from 0.60 to 0.84 (Table 4); text and audio safety evaluations were close (Not Unsafe 0.95 vs 0.93, Table 5).
• Preparedness ratings: cybersecurity low (CTF high school and college-level 19%, university-level 0%, professional-level 1%); biothreat low (tacit knowledge 69% consensus@10); model autonomy low (ARA 0%/100 trials); persuasion text modality borderline medium, speech low (AI audio clip effect size 78% of human), overall rating medium.
• Third parties: METR found no clear increase in autonomous capability relative to GPT-4; Apollo Research concluded GPT-4o is unlikely to cause catastrophic scheming.
• Authors' conclusions: Residual risks include decreased robustness under audio perturbations (anecdotal evidence), over-refusal in non-English, non-native accents, etc.; the authors plan continued monitoring and mitigation updates, and call for research on omni-modal adversarial robustness, anthropomorphism and emotional dependence.