OMNIGUARD: An Efficient Method for Cross-Lingual and Cross-Modal AI Safety Auditing

1. 01What problem does this paper attempt to solve?

   Train a classifier on internal representations to detect harmful prompts across languages and modalities.

   Harmful prompt detection is easily bypassed in low-resource languages, ciphers, and image and audio inputs.

   • Scenario: The authors believe that after large-scale deployment of LLMs, it is necessary to prevent both immediate harmful misuse and potential future harm. Defenses on high-resource languages (such as English) often detect harmful queries, but low-resource natural languages, ciphers, and image and audio inputs still often bypass them, i.e., jailbreaks.
   • Existing shortcomings: The authors adopt the view of Wei et al. (2023), attributing the cause to mismatched generalization in safety training: pretraining data is more diverse than safety fine-tuning, so safety capabilities do not generalize while general capabilities do. Standalone guard models such as LlamaGuard, AegisGuard, and WildGuard also require sending the input to a separate model.
   • Observation: The authors believe that LLMs/MLLMs form similar representations across languages and modalities at certain layers, and use a self-defined Universality Score (U-Score) to select these layers.
   • Method: Propose OMNIGUARD, which trains a lightweight harmfulness classifier on the embeddings of the selected layers; inference reuses representations already computed during generation, without modifying the original model. The authors state that it only applies to open-source models where embeddings can be read.
   • Threat model:
     • Defense goal: Determine whether the input is harmful, rather than rewriting the generative model's activations.
     • Attack inputs: Harmful prompts in low-resource natural languages, cipher languages, and image and audio inputs.
     • Victim system: The LLM or MLLM that accepts the query; the classifier is attached to its internal representations.
     • Defender assumptions: Requires access to the internal representations of that open-source model. The paper does not characterize the attacker's knowledge as white-box or black-box.

   Collapse this answer

2. 02What related research exists?

   Related work covers jailbreaks, multilingual and multimodal attacks, and guard models.

   Related Work is grouped by attack surface and review method; the quantitative differences are mainly written in the internal representation methods and multilingual guard.

   • English jailbreaks: Early work relied on handcrafted prompts (Shen et al., 2024b; Andriushchenko et al., 2024). This was followed by white-box gradient search (Zou et al., 2023b; Jones et al., 2023; Zhu et al., 2023), as well as genetic algorithms, fuzzing, and reinforcement learning to automatically generate prompts. There are also helper LLMs generating attacks (Chao et al., 2024; Ding et al., 2024; Mehrotra et al., 2024), few-shot in-context demonstrations, and multi-turn conversations (Russinovich et al., 2024; Li et al., 2024c).
   • Multilingual and cipher: Deng et al. (2024), Yong et al. (2024), Wang et al. (2024a) et al. write that attack success rates in low- and medium-resource languages are higher than in high-resource languages such as English. Jin et al. (2024), Jiang et al. (2024), Yuan et al. (2024a) use inserted cipher characters, ASCII art, or ciphers such as Morse, Atbash, and Caesar to bypass text defenses.
   • Multimodal jailbreaks: Liu et al. (2024b) use images related to harmful questions. Hu et al. (2025) argue that harmful images paired with seemingly benign text further increase the likelihood of jailbreak. Gong et al. (2025) and Wang et al. (2024b) typeset harmful queries into images. Audio jailbreaks are in Yang et al. (2025).
   • Review and internal representations: Intrinsic means are fine-tuning or RLHF; extrinsic means include perplexity filtering (Jain et al., 2023; Alon and Kamfonas, 2023; Hu et al., 2024) and separate guard models. Jailbreak Antidote (Shen et al., 2025) and AdaSteer (Zhao et al., 2025) directly modify internal representations; the paper says they only defend English text and take time comparable to the underlying LLM inference. DuoGuard uses two-player RL to generate multilingual data and fine-tunes Llama3.2-1B; PolyGuard fine-tunes Qwen-2.5-7B-Instruct with 17 languages and 1.91M samples. On the vision side there are LlamaGuard3 Vision, VLMGuard, LLavaGuard. The authors say that, to their knowledge, OMNIGUARD is the first guard model for harmful audio input.
   • Baselines and benchmarks: Text comparisons are LlamaGuard 1/2/3, AegisGuard (Permissive/Defensive), WildGuard, HarmBench's llama and mistral classifiers, MD-Judge, DuoGuard, PolyGuard; the latter two are also LoRA fine-tuned with the same 37 languages as this paper. Vision comparisons are Llama Guard 3 Vision, VLMGuard, LLavaGuard. SpeechGuard only adds noise and does not classify, so it was not used as an audio baseline. Benchmarks are in Table 1, including MultiJail, Xsafety, RTP-LX, HarmBench, AdvBench, Hades, FigStep, MM SafetyBench, AIAH, VoiceBench.

   The authors position OMNIGUARD as a standalone content safety classifier: it discriminates using representations shared across languages and modalities, without rewriting activations; in the conclusion they say its accuracy and training and inference overhead are better than standard guard models.

   Collapse this answer

3. 03How does the paper address this problem?

   Use U-Score to select layers that are more aligned across languages or modalities, then train a lightweight classifier.

   OMNIGUARD does not train a separate guard LLM, but instead finds layers in the generation model that are aligned across languages or modalities, and uses the representations from those layers to train a lightweight harmfulness classifier.

   Setup and representation

   • Input: user prompt, which can be natural language or cipher, with modality of text, image, or audio.
   • Representation: the representation of a prompt at a given layer is the average of the representations of all tokens in that prompt.
   • Output: a binary classification of harmful or benign. The original model's parameters are not updated; only the classifier is trained.

   Layer selection with U-Score

   • U-Score is the average cosine similarity of translated sentence pairs minus the average cosine similarity of non-translated sentence pairs: U-Score:=1/N∑_iCosSim(Emb(e_i),Emb(l_i))−1/(N(N−1))∑_{i≠ j}CosSim(Emb(e_i),Emb(l_j)). e_i and l_i are an English sentence and its translation, and Emb is the representation at a given layer.
   • When switching modalities, replace the translation embedding with an embedding from another modality, such as a caption paired with an image, or a transcription paired with audio.
   • Text: 53 natural languages from Flores200, plus 20 ciphers; computed at each layer of Llama3.3-70B-Instruct and averaged over languages. Natural languages are higher in middle layers, with the main text stating layer 57 (out of 81 layers) is highest. Cipher U-Scores are much lower, and the authors believe the model fails to represent semantic similarity in these languages.
   • Image and audio: MM-Vet v2 has 517 entries, captions generated by Molmo-7B, and embeddings extracted with Molmo-7B. The main text states the peak is at layer 22 (out of 28 layers), while the Figure 2 note says layers 21–25 are highest. Audio uses Alpacaeval from VoiceBench, 636 pairs, with model LLaMA-Omni 8B; the main text states the peak is at layer 21 (out of 32 layers), while the figure note says layers 20–23. The y-axis label is Universality Score (U-Score), and the figure note calls the latter two curves Cross-Model Alignment Score.

   Classifier and inference

   • Take the embedding from the layer with the highest U-Score and train an MLP: two hidden layers with widths 512 and 256. The paper does not specify the optimizer, epochs, or learning rate.
   • During generation, feed the embedding already computed at that layer into the classifier. The authors state the only extra computation comes from this MLP. Closed-source models cannot access embeddings, so this does not apply.

   Three training data types

   • Text: layer 57. Randomly sample 2,800 entries from Aegis AI Content Safety, balanced between harmful and benign. Translate into the other 52 natural languages (Google Translate API) and 20 ciphers (fixed rules), for 73 languages total; only half are used for training, listed in Appendix A, and the main text calls it 37 languages. The authors state this set is about 18× smaller than baseline training data; the results section separately writes OMNIGUARD 103K entries and PolyGuard 1.91M entries.
   • Vision: layer 22. Harmful samples are 2,000 image-query pairs from JailBreakV-28K, plus 1,024 pairs from VLSafe; benign samples are 517 pairs from MM-Vet v2. The authors state the training amount is about 3,500, while LlavaGuard is about 5,500.
   • Audio: layer 21. Use the English portion of the text setup training data, converted with Kokoro text-to-speech. The paper does not separately list the number of audio training set entries.

   Collapse this answer

4. 04What experiments did the paper conduct?

   Multilingual average 86.36%, image-text average 95.44%; MML Base64 only 0.40%.

   Experimental Setup

   • Backbone: The main text experiment uses layer 57 of Llama3.3-70B-Instruct. Table B3 additionally uses Llama3.1-8B-Instruct, Gemma-3-4B-Instruct, Qwen-3-4B-Instruct, Olmo-2-7B-Instruct, Mistral-8B-Instruct, each taking the layer with the highest U-Score for that model. Vision uses Molmo-7B layer 22, audio uses LLaMA-Omni 8B layer 21. Audio is synthesized from text by Kokoro.
   • Training scale: Text has 2,800 balanced samples, translated into 37 training languages. Vision harmful samples are 2,000+1,024, benign 517. The code attack experiment is trained only on the English subset. The authors say PolyGuard is 1.91M samples and OMNIGUARD is 103K samples.
   • Baselines: The guards in Table 2, plus MD-Judge in the result tables. DuoGuard and PolyGuard are LoRA fine-tuned with the same 37 languages, all linear layers, rank 8, alpha 16, 1 epoch, learning rate 2e-4. VLMGuard uses the same vision training data as OMNIGUARD. The paper says there is no existing harmful audio classification model.
   • Data: Table 1. The multilingual set includes MultiJail (315), Xsafety (28,000), RTP-LX (30,300), AyaRedTeaming (2662), Thai Toxicity tweets (3,300), Ukr Toxicity (5,000). The remaining English text is translated into 72 other languages (52 natural languages, 20 ciphers). Vision includes Hades (750), FigStep (500), MM SafetyBench (1680), etc. Audio includes AIAH (350) and text benchmarks synthesized by Kokoro.
   • Metrics and hardware: Harmfulness classification accuracy. Inference time is measured on 1 L40, 4 CPUs, 50 GB RAM on AdvBench in English, Spanish, French, Telugu, and base64. Few-shot averages over 50 random example sets, with error as standard error of the mean.
   • Judgment: Relative to dataset labels. The paper does not write an LLM judge, classification threshold, or report the number of repetitions for the main tables.

   Main Results

   The table below shows the average accuracy (%) printed in each table. Fine-tuning results for unseen languages are not merged into this table.

   Swipe horizontally to read the table

   Method	Multilingual Avg.	Translated text Avg.	Image+text Avg.	Typography image Avg.
   OMNIGUARD	86.36	84.85	95.44	79.76
   PolyGuard	83.19	66.07	not reported	not reported
   AegisGuard (D)	80.62	80.28	not reported	not reported
   LlamaGuard 3	56.52	62.98	not reported	not reported
   HarmBench (llama)	14.92	14.12	not reported	not reported
   VLMGuard	not reported	not reported	88.74	42.84
   Llama3 Vision GRD	not reported	not reported	47.36	74.36
   LLavaGuard	not reported	not reported	18.46	2.28

   From Table 3(A)(B), Table 4(A)(B). The table omits LlamaGuard 1/2, AegisGuard (P), WildGuard, HarmBench (mistral), MD-Judge, DuoGuard; DuoGuard's multilingual average is 54.56%.

   • The authors say the average is highest in the multilingual and translated settings, and call MultiJail, RTP-LX, Aya RedTeaming new SOTA. In Table 3(A), PolyGuard is higher on Xsafety, Thai Tox, Ukr Tox (96.41%, 70.43%, 76.07%); in Table 3(B), PolyGuard is higher on TJS and ToxText, and AegisGuard (D) is 90.77% on AdvBench.
   • The authors say the grouped average is highest for both types of image benchmarks, and say what could not be detected is MML Base64 (0.40%; Llama3 Vision GRD is 98.80%). On RTVLM, VLMGuard is 94.00% and OMNIGUARD is 89.00%.
   • The abstract writes 11.57% higher than the strongest baseline for multilingual and 20.44% higher for image prompts, which is not the same number as the Avg. difference in the table above; the paper does not write the calculation basis. The introduction also writes 86.22% average for 53 natural languages, 88.31% for images, 93.09% for audio; 88.31% is not the grouped average of Table 4, and Table 5 does not print an overall average.

   Unseen Languages, Audio, and Few-shot

   • Unseen languages: Table 3(C) averages are OMNIGUARD 81.88%, FT PolyGuard 75.13%, FT DuoGuard 37.1%. On TJS, FT PolyGuard 74.07% is higher than 72.82%, and on ToxText 72.33% is higher than 71.44%.
   • Audio: In Table 5, OMNIGUARD ranges from 82.03% on TJS to 98.85% on AdvBench; AIAH is 91.14%. The authors say audio is similar to or higher than English text. In comparisons, FQ audio is 90.42% and English text 93.3%; TJS audio is 82.03% and English text 93.2%. LlamaGuard3 only reports English text.
   • Code few-shot: Three CodeAttack formats, trained only on the English subset. The authors say it is near 50% with no examples and near 100% with fewer than five examples; baselines rise more slowly. Averaged over 50 random example sets. The text says guards that already reach 90% or more with zero-shot are omitted. Figure 3 does not give endpoints in table form.

   Layer Selection, Speed, and Language

   • Table 6 (Thai Tox, Ukr Tox, TJS, ToxText): layer 57 averages 73.4, layer 10 is 64.42, layer 75 is 68.26, last layer 59.05. The authors say it improves by 5% to 14%. In Table B2, layer 55 averages 73.06, close to layer 57's 73.40.
   • Table 7: OMNIGUARD 0.04 seconds, DuoGuard 4.85 seconds, PolyGuard 409.90 seconds. The authors say it is about 120× faster than the fastest baseline DuoGuard.
   • Table B1: high-resource PolyGuard 94.47%, OMNIGUARD 88.25%; low-resource 85.56% vs PolyGuard 79.22%; on cipher, AegisGuard (D) 83.21%, OMNIGUARD 73.06%, PolyGuard 21.28%. Figure 4 annotates R^2 = 0.24. The authors say harmfulness accuracy is broadly correlated with zero-shot sentiment classification, with natural languages higher than ciphers; when sentiment classification is near 50%, harmfulness accuracy can still be high. Sentiment classification uses Llama3.3-70B-Instruct, with data being 72 translations of SST-2.

   Other Ablations and Analysis

   • Table B3 averages: Llama3.3-70B-Instruct 84.85, Mistral-8B-Instruct 82.81, Llama3.1-8B-Instruct 82.16, Olmo-2-7B-Instruct 81.99, Gemma-3-4B-Instruct 76.60, Qwen-3-4B-Instruct 76.11. The authors say smaller LLMs have lower averages.
   • Table B2: layers 58–62 average between 71.50 and 72.82, lower than the selected layer 57's 73.40.
   • Figure 2: natural language U-Score is higher in middle layers; the cipher curve is lower. The peak layers for images and audio are in question 3.

   Collapse this answer

5. 05What are some points that can be further explored?

   The authors state it depends on whether the underlying model understands the input, and requires internal representations.

   Limitations and Future Directions Noted by the Authors

   • Depends on whether the underlying model understands the input: If the model does not understand that language, image, or audio, OMNIGUARD may not detect harmfulness. The authors say this is not a unique limitation of theirs (Limitations).
   • Shared representations are an empirical observation: The authors say they found universally similar representations on the models and modalities used, but did not exhaustively check all models; the assumption may not hold on unused models (Limitations).
   • Requires internal representations: Therefore it does not apply to closed-source models (Limitations).
   • Evaluation sets are fixed benchmarks: Results are based on a set of evaluation datasets commonly used in the field; real-world performance may differ (Limitations).
   • The classifier can fail: Ethics states it is not a perfect classifier, and unexpected failures may still allow harmful misuse of the LLM.

   Limitations and Conclusion do not list an additional experimental future work list.

   Experimental Coverage

   • The main text result backbone is Llama3.3-70B-Instruct; Table B3 additionally reports Llama3.1-8B-Instruct, Gemma-3-4B-Instruct, Qwen-3-4B-Instruct, Olmo-2-7B-Instruct, Mistral-8B-Instruct.
   • Languages are 53 natural languages and 20 ciphers from Flores200; 37 of them are used for training, and Table 3(C) reports unseen languages.
   • The vision backbone is Molmo-7B, the audio backbone is LLaMA-Omni 8B. The paper states that SpeechGuard only adds noise and does not classify harmfulness, so it was not used as an audio classification baseline; audio comparisons are OMNIGUARD and LlamaGuard3 on English text.
   • The main metric is accuracy relative to dataset labels. The paper does not report the number of repetitions for the main tables, nor agreement with human judgment or classification thresholds.
   • Few-shot covers three CodeAttack formats with 50 random example sets; the classifier is trained only on the English training subset.

   Collapse this answer

6. 06Summarize the main content of the paper

   OMNIGUARD detects harmful prompts on shared internal representations and reuses generation embeddings.

   OMNIGUARD is a harmful prompt classifier attached to the internal representations of open-source LLM/MLLM, targeting languages and modalities that safety training does not generalize to.

   • Problem: Defenses on high-resource languages often block harmful queries; low-resource natural languages, ciphers, images, and audio still often bypass them. The authors attribute this to mismatched generalization.
   • Method: U-Score uses cosine similarity of mutually translated or cross-modal pairs, minus the similarity of non-paired items, to select layers with stronger alignment. The classifier is a two-layer MLP with widths 512 and 256, input being the token-averaged embedding of that layer. Text uses layer 57 of Llama3.3-70B-Instruct, images use layer 22 of Molmo-7B, audio uses layer 21 of LLaMA-Omni 8B. Inference reuses the embeddings from generation and does not update the original model.
   • Text: Table 3(A) multilingual average is 86.36%, with PolyGuard at 83.19% in the table; PolyGuard is higher on Xsafety, Thai Tox, Ukr Tox. Translated text average is 84.85%, unseen language average is 81.88%. In Table B1, PolyGuard is 94.47% on high-resource and AegisGuard (D) is 83.21% on cipher, respectively higher than 88.25% and 73.06%.
   • Modalities and time: Image+text pairs average 95.44%, typography images average 79.76% (Table 4); MML Base64 is 0.40%. The introduction writes audio accuracy of 93.09%. In Table 7, average inference time on five AdvBench languages is 0.04 seconds, which the authors say is about 120 times faster than the fastest baseline DuoGuard.
   • Authors' conclusion: The authors say the method can detect harmful prompts across languages and modalities, with training and inference more economical than standard guard models; they also state it depends on whether the underlying model understands the input and cannot be used on closed-source models. The 11.57% and 20.44% in the abstract are not written with the calculation corresponding to the table Avg. in the paper.

   Collapse this answer

Related papers