NYUAD participates in the ArGuard shared task: a multimodal embedding model for detecting Arabic hateful memes and unsafe prompts
NYUAD at ArGuard Shared Tasks: Multimodal Embedding Models for Detecting Arabic Hateful Memes and Unsafe Prompts
The authors use pretrained and LoRA fine-tuned multilingual embedding models connected to LR/RF classifiers to detect Arabic hateful memes and unsafe prompts, and report accuracy and F1 on the ArGuard 2026 tasks.
The paper does not provide a threat model from the attacker's perspective; the task is defensive content moderation: classifying Arabic memes as hateful/not-hateful, and classifying user prompts as safe/unsafe and into seven fine-grained categories.
- Arabic content moderation resources are insufficient, with many dialects and few datasets, requiring automatic methods to identify hateful memes and unsafe prompts.
- Use multilingual embedding models pretrained or fine-tuned with LoRA to extract text/image vectors, then connect to logistic regression or random forest classification; in the blind test phase, switch to cosine distance-based kNN for safe/unsafe.
- On the development set, fine-tuned Gemma 3 (mean pooling) had accuracy 0.79 on the meme task; OpenAI text-embedding-3+LR had accuracy 0.96 on safe/unsafe and 0.86 on seven-way classification; in the blind test phase, meme macro F1 0.7707, safe/unsafe kNN macro F1 0.8449.
- The authors point out that data annotation subjectivity introduces inconsistency, Arabic dialects remain a bottleneck, and models lack local cultural knowledge; the experiments were conducted only on the ArGuard 2026 dataset, and the paper does not report the number of repetitions or statistical significance tests.
- Evaluated models
- Google Gemini embedding 2OpenAI text-embedding-3-largeOmartificial-Intelligence-Space/Arabic-Triplet-Matryoshka-V2unsloth/gemma-3-4b-ptunsloth/gemma-4-E4B-itgoogle/gemma-4-E2B
- Benchmarks
- ArabicNLP ArGuard 2026 challenge
- Metrics
- AccuracyMacro F1-scorePrecisionRecall
In the ArabicNLP ArGuard 2026 shared task, the NYUAD team used a pretrained and fine-tuned multilingual embedding model to detect Arabic hateful memes and unsafe prompts. The model learns semantic representations from text and visual content, and evaluation on the challenge dataset shows that LLM-based embedding models perform better. The work aims to improve content moderation and create a safer, more trustworthy online environment for Arabic-speaking users.
Paper analysis
What problem does this paper attempt to solve?
Detecting Arabic hateful memes and unsafe prompts using embedding models plus traditional classifiers, covering the three ArGuard 2026 tasks.
Core problem: how to automatically identify Arabic hateful memes (hateful/not-hateful) and unsafe user prompts (safe/unsafe and seven fine-grained categories).
- Scenario and importance: The authors state that social media and generative AI have increased Arabic text, memes and prompts, and that such content may be used to spread hate speech and harmful instructions; Arabic has about 380 million speakers and includes six major dialect groups, and content moderation resources are insufficient.
- Shortcomings of existing methods: The authors state that English content moderation has made more progress, while Arabic is limited by dialect diversity, a shortage of high-quality training data and a lack of tools.
- Core idea: Use pretrained and fine-tuned LLM embedding models to learn textual and visual semantic representations, then connect them to traditional machine learning classifiers, which suits low-resource, dialect-diverse Arabic content.
- What the paper proposes: Participation in the three tasks of the ArabicNLP ArGuard 2026 challenge—meme binary classification, prompt safe/unsafe binary classification, and prompt seven-way classification (adult content, hate speech, bullying, harassment, fraud & deception, harm to others, self-harm)—and proposes two research questions (RQ1: whether pretrained embeddings + ML classifiers are effective; RQ2: whether fine-tuned embeddings can improve accuracy).
- Threat model: The paper does not provide a threat model from the attacker's perspective; this is a defensive content moderation task.
- Victim system: content moderation scenarios for Arabic-speaking users.
- Detection targets: memes (image + Arabic text) and user prompts.
- The paper does not state attacker goals, knowledge or capabilities.
What related research exists?
Related work is divided into two groups: harmful prompt detection and Arabic hate meme detection; this paper positions itself as applying embedding models to the three tasks of ArGuard 2026.
Harmful prompt detection
- Commercial moderation APIs: The paper mentions moderation tools from Azure, OpenAI, and Alibaba, stating that they generally combine rule-based blocking with machine learning models trained on safety benchmarks (Yang et al., 2025).
- Llama Guard series (Inan et al., 2024): Trains a language model on large-scale annotated data to determine whether an input prompt or model response violates predefined safety policies.
- Subcategory prompt classification (Gupta et al., 2024): Identifies specific harm types in LLM inputs and uses explainable AI.
- Arabic prompt detection (Mubarak et al., 2025): Builds a 24K Arabic safety dataset (12K naturally human-written + 12K synthetic harmful prompts generated by GPT-4o) and benchmarks Arabic and multilingual models in the 7B to 13B parameter range.
Arabic hateful meme detection
- Multimodal analysis (Alam et al., 2024): Fuses AraBERT text features and ConvNxT image features to detect hate in Arabic memes.
- Embedding model comparison (AlDahoul and Zaki, 2025b): Tests base, fine-tuned, and pretrained embedding models for detecting meme hate; the authors state that the fine-tuned model (specifically Gemini 2.5 Flash) performed better.
- Fusion and ensemble architectures (Zaytoon et al., 2025; El Attar, 2025; Safwan, 2025): Include cross-attention models, progressive CNN fusion models, and two-stage models with custom-trained embeddings plus gated fusion classifiers.
Baselines and benchmarks: The paper does not set up an independent baseline method; the comparison objects are multiple embedding models (Google Gemini embedding 2, OpenAI text-embedding-3-large, Arabic-Triplet-Matryoshka-V2, fine-tuned Gemma 3/Gemma 4) and two classifiers (LR, RF), with data from the ArabicNLP ArGuard 2026 challenge (Alam et al., 2026; Kmainasi et al., 2026).
The authors position this paper as applying LLM embeddings (including LoRA fine-tuning) with LR/RF classifiers to the meme and prompt detection tasks of ArGuard 2026.
How does the paper address this problem?
Extract embedding vectors and connect to LR/RF classifiers; fine-tune Gemma 3/4 for the meme task and Gemma 4 for the prompt task; switch to kNN for safe/unsafe in the blind test phase.
Overall approach: Use pretrained or LoRA fine-tuned multimodal/text embedding models to encode memes (image+text) or prompts into dense vectors, then connect them to logistic regression (LR) or random forest (RF) classifiers, tuning parameters on the development set to select the best combination.
Data
- Meme dataset: Images plus extracted Arabic text; 3,500 training, 500 development, 500 test (released in the development phase), and another 500 blind test set in the test phase.
- Prompt dataset: 16,831 training (13,436 unsafe, 3,395 safe), 2,406 development (1,930 unsafe, 475 safe), 4,810 blind test; the seven-way training set class counts are adult content 4,592, harm to others 3,039, self-harm 2,722, harassment 2,254, fraud and deception 1,764, bullying 1,517, hate speech 943; safe or unsafe prompts may both carry the four labels self-harm, adult content, harassment, and harm to others.
Meme hate detection
- Pretrained embeddings: Google Gemini embedding 2 (multimodal input, 3072 dimensions), OpenAI text-embedding-3-large (text only, 3072 dimensions), Arabic-Triplet-Matryoshka-V2 (text only, 768 dimensions, fine-tuned based on aubmindlab/bert-base-arabertv02).
- Fine-tuned embeddings: Use Unsloth's SFT trainer and UnslothVisionDataCollator to fine-tune unsloth/gemma-3-4b-pt and unsloth/gemma-4-E4B-it; batch size 8, gradient accumulation 2 steps (effective batch 16), fused AdamW, 3 epochs, cosine learning rate peak 0.0002, 3% warmup, gradient clipping 0.3.
- LoRA settings: Apply LoRA via FastVisionModel, enabling all four fine-tuning flags, targeting "all-linear" modules; r=lora_alpha=16 for Gemma 3, r=lora_alpha=32 for Gemma 4, dropout 0; vision_layers, language_layers, attention_modules, mlp_modules all set to True.
- Vector aggregation: Two types, mean pooling and last-token pooling; both Gemma 3 and Gemma 4 generate 2560-dimensional vectors; embeddings connect to LR, tuning LR's inverse regularization strength C.
Prompt detection
- Embedding models: OpenAI text-embedding-3, Arabic-Triplet-Matryoshka-V2, google/gemma-4-E2B (text input); when fine-tuning Gemma 4, use 4-bit NF4 quantization (double quantization), batch size 4, fused AdamW, 3 epochs, LoRA r=lora_alpha=16; Gemma 3/4 generate 1536-dimensional vectors.
- Classifiers: safe/unsafe uses LR; seven-way classification uses LR and RF, tuning RF's number of trees, max depth, min samples split, and using weighted classes to handle class imbalance.
Distance method in the blind test phase
- In response to the distribution change in the safe/unsafe blind test set, the authors trained a kNN: using only the L2-normalized embeddings of known unsafe training samples from OpenAI Text-Embedding 3, computing the mean cosine distance between a test sample and its 5 nearest unsafe neighbors (k=5), with threshold 0.45; a distance greater than or equal to the threshold is classified as safe, otherwise as unsafe.
Evaluation method
- Metrics are accuracy, macro F1, precision, recall; development-phase results are generated on the labeled development set, and embedding methods are also compared on the unlabeled test data released in the development phase; final evaluation is on the blind test set. The paper does not report the number of repetitions.
What experiments did the paper conduct?
For the three tasks, C and RF hyperparameters were tuned on the development set, and unlabeled test data and the blind test set were also tested; meme macro F1 was highest at 0.77, and prompt safe/unsafe was highest at 0.96.
Experimental setup
- Embedding models tested: For the meme task, Google Gemini embedding 2, OpenAI text-embedding-3-large, Arabic-Triplet-Matryoshka-V2, fine-tuned Gemma 3 (mean/last token pooling), fine-tuned Gemma 4 (mean/last token pooling); for the prompt task, OpenAI text-embedding-3, Arabic-Triplet-Matryoshka-V2, fine-tuned Gemma 4 (mean/last token pooling).
- Datasets: ArGuard 2026 meme dataset (train 3,500, dev 500, test 500, plus 500 blind test) and prompt dataset (train 16,831, dev 2,406, blind test 4,810).
- Classifiers and tuning: LR tunes C (0.001 to 1000); RF tunes number of trees (100/300/700), max depth (7/20), min samples split (10/2).
- Metrics: accuracy, macro F1, precision, recall; the classification report gives support per class.
- Evaluation method: Uses the datasets' own labels, no LLM evaluation; the paper does not report the number of repetitions or statistical significance tests.
- Result sources: Development-phase results come from the labeled development set; there is also a comparison on unlabeled test data in the development phase and final blind test phase results.
Main results
Accuracy / macro F1 of the best combination for each task on the development set (according to Table 5, Table 6, Table 8, each taking the best value in each column):
Swipe horizontally to read the table
Task Best combination Accuracy Macro F1 Meme binary classification Fine-tuned Gemma 3 (mean pooling)+LR, C=0.001 0.79 0.77 safe/unsafe OpenAI text-embedding-3+LR, C=0.01 0.96 0.93 Seven-way classification (LR) OpenAI text-embedding-3+LR, C=0.001 0.86 0.85 Seven-way classification (RF) OpenAI text-embedding-3+RF, depth 20, 700 trees, split 10 0.81 0.79 - The authors state that in the meme task, fine-tuned Gemma 3 (mean pooling) outperformed Gemma 3 last token, fine-tuned Gemma 4, and pretrained embeddings (Table 4, Table 5); among pretrained embeddings, OpenAI text-embedding-3 had the highest accuracy 0.73 (C=0.001, Table 4).
- The authors state that comparing Table 7 and Table 8, LR outperforms RF on all embedding models, and believe that the relationship between embedding features and the target may be close to linear in high-dimensional space.
- On unlabeled test data in the development phase: Gemma 4 (last token) had the highest accuracy 0.754 and F1 0.729; Gemma 3 (mean) 0.744/0.717; Gemma 4 (mean) 0.734/0.699; Gemma 3 (last token) 0.730/0.688; OpenAI Embedding 3 was lowest, 0.696/0.658.
Blind test phase results
- Meme classification: Gemma 3 embedding+LR, macro F1 0.7707, precision 0.7724, recall 0.7691, accuracy 0.8100.
- safe/unsafe: kNN (L2-normalized OpenAI Text-Embedding 3 embeddings, k=5, threshold 0.45), macro F1 0.8449, precision 0.86124, recall 0.8456, accuracy 0.8468.
- Seven-way classification: OpenAI Text-Embedding 3+LR, macro F1 0.7234, precision 0.7337, recall 0.7295, accuracy 0.7377.
Error analysis
- Distribution shift: The authors state that the blind test set has a different distribution from the training/validation sets—the training and validation sets are mostly unsafe, while the blind test set is larger and half safe, causing models with high scores on the validation set to drop in performance on the blind test set, so the above kNN method was adopted.
- Seven-way classification weaknesses: According to Table 3 and Figure 10, the harm-to-others class had the lowest F1 at 0.74; the authors state that the model often misclassifies content that actually belongs to bully or self-harm as harm-to-others, and believe the reason is that harm-to-others has a broad semantic range and overlaps with bullying, harassment, and fraud.
Other ablations and analyses
- Effect of C (memes, Table 4, 5): Pretrained embeddings were best at C=0.001, and performance decreased as C increased; Google Embedding 2's LR did not converge at C=100 and 1000 (marked as x in the table).
- Effect of C (prompts, Table 6, 8): On safe/unsafe, each embedding was best at C=0.001–0.01; the seven-way classification showed the same trend.
- RF hyperparameters (Table 7): max depth 20 and 700 trees were generally better than depth 7 and 100 trees.
- t-SNE (Figure 12): The hateful and not-hateful embedding points of memes were scattered interleaved and did not form separable clusters; based on this, the authors believe that this model's embeddings cannot distinguish the two classes in low-dimensional space.
- t-SNE (Figure 11): The safe and unsafe embeddings belonged to different regions, but some points overlapped and were not completely separated by a margin.
What are some points that can be further explored?
The authors point out three limitations: annotation subjectivity, dialects, and cultural knowledge; the experiments cover only ArGuard 2026 data and the listed embedding models.
Limitations and future directions noted by the authors
- Annotation subjectivity (Limitations section): The authors state that data annotation is highly subjective, different annotators interpret the same content differently, introducing inconsistency into the training set and potentially reducing overall model performance.
- Dialect issue (Limitations section): The authors state that the variability of Arabic dialects remains the biggest technical bottleneck; models can generally read Modern Standard Arabic, but internet memes are almost always written in colloquial dialects.
- Lack of cultural knowledge (Limitations section): The authors state that even after fine-tuning, models lack local cultural knowledge because they are mainly trained on Western data, making it difficult to recognize regional accents, local internet trends, or how Arab public figures are used in memes, and that they are often ineffective against implicit hate speech involving visual inside jokes and cultural coding.
- The paper does not state an explicit future work plan.
Scope of experimental coverage
- There are 6 embedding models tested: Google Gemini embedding 2, OpenAI text-embedding-3-large, Arabic-Triplet-Matryoshka-V2, unsloth/gemma-3-4b-pt, unsloth/gemma-4-E4B-it, google/gemma-4-E2B (including their fine-tuned versions).
- Data come only from the ArabicNLP ArGuard 2026 challenge's meme dataset (train 3,500, dev 500, test 500, blind test 500) and prompt dataset (train 16,831, dev 2,406, blind test 4,810).
- The classifiers are logistic regression, random forest, and kNN in the blind test phase (k=5, threshold 0.45).
- The paper does not report the number of repetitions, statistical significance tests, or human evaluation.
- The paper does not report training time, inference cost, or ranking comparison with other participating systems.
Summarize the main content of the paper
Embedding models + LR/RF detect Arabic hateful memes and unsafe prompts, with development-set accuracy up to 0.96 and blind-test macro F1 up to 0.8449.
This is a participation report for the ArabicNLP ArGuard 2026 shared task, whose goal is to use embedding models to detect Arabic hateful memes and unsafe user prompts.
- Problem: Arabic content moderation resources are insufficient, and dialects are diverse; the tasks include meme binary classification (hateful/not-hateful), prompt safe/unsafe binary classification, and seven-way classification.
- Method: Use pretrained embeddings (Google Gemini embedding 2, OpenAI text-embedding-3-large, Arabic-Triplet-Matryoshka-V2) and LoRA fine-tuned Gemma 3/Gemma 4 to extract vectors, connect them to LR or RF classifiers, and tune parameters on the development set.
- Meme task: On the development set, fine-tuned Gemma 3 (mean pooling)+LR was best, with accuracy 0.79 and macro F1 0.77 (Table 5); in the blind test phase, macro F1 0.7707 and accuracy 0.8100.
- safe/unsafe: On the development set, OpenAI text-embedding-3+LR accuracy 0.96 (Table 6); the authors state that a distribution shift in the blind test set (half are safe) caused the original model's performance to drop, and after switching to kNN (k=5, threshold 0.45), the blind test macro F1 was 0.8449 and accuracy 0.8468.
- Seven-way classification: On the development set, OpenAI text-embedding-3+LR accuracy 0.86 and macro F1 0.85 (Table 8), better than RF (highest 0.81/0.79, Table 7); blind test macro F1 0.7234; the harm-to-others class was weakest (F1 0.74), which the authors attribute to its semantics overlapping with other categories.
- Authors' conclusions: The authors believe that LLM embedding models (especially after fine-tuning) can improve Arabic content detection performance, and that fine-tuned Gemma 4 is a freely deployable open-source alternative; they also point out that annotation subjectivity and lack of dialect and cultural knowledge are the main limitations.