MemeTrace: A Multimodal LLM Agent for Safety-Aware Meme Origin Analysis

Platform moderation of memes lacks provenance on source, spread, cultural symbolism, and social risk; a single-image safe/unsafe classification is insufficient.

• Scenario: The authors state memes change quickly with news and depend on cultural context; humor, irony, and sarcasm can mask intent. The authors use a face-morphing image as an example of how conspiracy theories can hide in visual suggestions. Unsafe content involved includes hate ideologies, misinformation, propaganda, and conspiracy theories.
• Existing gaps: Prior work mostly does closed-set classification, mainly looking at surface visuals or overlaid text, without explaining why something is harmful, and lacking recent events and platform usage; the authors state this makes generalization to new memes difficult.
• Authors' framing: Determining harm requires provenance, i.e., data source, reuse history, and other context. Static VLMs cannot incorporate recent knowledge and can only recognize outdated surface elements.
• Proposed here: The authors formalize Meme Provenance, generating an 11-field report from a single image; and present the agent MemeTrace, a 100-item Meme100, and automatic quality metrics computed from report text.

Closed-set harmful meme classification

• SemEval Memotion (Sharma et al., 2020) fuses CNN image features with BiLSTM text features to predict meme sentiment or whether it contains offensive content.
• MOMENTA (Pramanick et al., 2021b) fuses image and overlaid-text features for harmful meme detection.
• The introduction also lists Lippe et al. (2020), Zhou and Chen (2020), Pramanick et al. (2021a), Kiela et al. (2020a,b), Muennighoff (2020), Cao et al. (2022, 2023) as similar classification studies. The authors state such methods only look at surface elements and generalize poorly to new memes lacking recent context.

Lineage, evolution, and ideology

• Zannettou et al. (2018), Ling et al. (2021) do meme lineage or origin; Qu et al. (2023a) do evolution analysis.
• Bi et al. (2023), Ma et al. (2025), Lin et al. (2024b) focus on sociopolitical ideologies represented by memes. Related work also places Qu et al. (2023b) in this citation group, but the named evolution work is 2023a.
• The authors state these works still rely on static multimodal models, analyzing one dimension at a time, and cannot incorporate real-time online evidence.

LLM agents for meme analysis

• Lin et al. (2024a), Liu et al. (2025), Huang et al. (2024, 2025) are grouped together: using LLM agents for meme analysis. The authors state these methods are still limited to single-meme classification without systematically modeling broader context.

Baselines and evaluation data

• The main baseline is GPT-4o: same report prompt as MemeTrace but without external evidence. Appendix J also has GPT-4o native web search, same backbone and prompt, relying only on built-in text search. The evaluation set is the authors' constructed Meme100.

The authors position this paper as: using dynamic external knowledge and step-by-step reasoning to decompose a meme into an 11-field provenance report, rather than static closed-set classification.

MemeTrace is a multimodal LLM agent: instead of a single VLM inference, it uses a Think–Act–Evaluate loop to coordinate perception, retrieval, evidence evaluation, and report synthesis. Figure 2 shows the closed loop of six modules.

Task and Report Fields

• Input is a single meme image. Output is a structured report with 11 fixed fields in Section 2 and Appendix A's Table 5.
• The five groups are: Semantics (About, Structure), Spread (Origin, Spread), Culture (Culture, Usage, Target Audience), Psychological Impact (Emotional Impact, Cognitive Load, Persuasive Technique), and Safety (Safety Flag).
• Safety Flag is chosen from a fixed set and can be multi-labeled: abusive, hateful, misogynous, cyberbullying, disinformation, fake misattribution, propaganda, offensive language, derogatory, trolling, sarcasm, unethical, political, satire, sensitive.
• About, Origin, and Spread follow KnowYourMeme entry structures; the other fields are extended by the authors to write cultural context, usage, and risks.

Perception and Local Retrieval

• OCR: EasyOCR extracts English text. The authors state raw output often contains irrelevant content like watermarks, so GPT-4o verifies, corrects errors, and retains the meme's core text.
• MemeDB: A local store of KnowYourMeme confirmed entries. Appendix B states 21,305 entries as of June 2025; the main text also writes 21K+. Each entry includes identifier, text, time, source, region, and tags.
• Image and text metadata are encoded with CLIP and concatenated; FAISS retrieves top-3. The authors state CLIP neighbors may be visually similar but semantically unrelated, so GPT-4o scores 0–10 semantic relevance, keeping only candidates above 8 and their metadata. Appendix B states the local library speeds retrieval and avoids repeated searches for similar memes.

Two External Retrieval Paths

• RIS: Uses Google reverse image search to fetch relevant web pages. Domain scores are ranked by a confidence lower bound plus a decayed exploration bonus, and only the top-5 are crawled to avoid irrelevant URLs. GPT-4o's RIS evidence checker cleans webpage noise, generates summaries with inline citations, and assigns each source a usefulness score of 0–1, which is written back to the domain score.
• Tavily has two modes. Element-Extraction Mode performs broad retrieval when RIS evidence is insufficient: it extracts salient visual and textual elements, especially proper nouns like people, organizations, and places, forms queries from the top two, and retrieves up to 5 results with short summaries and links. Evaluator-Query Mode, when some fields remain uncertain, incomplete, or inconsistent, has the confidence evaluator write queries based on the current evidence state. A quality checker also assigns 0–1 scores.

Routing and Report Synthesis

• The confidence evaluator judges after each tool call whether evidence is sufficient or conflicting, and chooses to continue retrieval, modify the query, or synthesize the report. Figure 3's output includes new_confidence_score, next_action, justification, and next_query.
• The order is: local library first; if insufficient, RIS; if still insufficient, Tavily's Element-Extraction; if still insufficient, Evaluator-Query, which rewrites keywords using history from the local library, RIS, and Tavily and iterates. It stops when evidence is sufficient or the iteration limit is reached. The default limit is 5. Reports are also synthesized when the limit is reached.
• Report synthesis uses GPT-4o and a dedicated prompt, integrating collected evidence according to the Section 2 schema. The main experiment uses temperature 0 via the OpenAI API. Implementation uses LangGraph; domain scores and MemeDB metadata are stored in SQLite, and execution logs in Markdown.
• When replacing the backbone, the iteration limit, temperature, and prompt templates remain unchanged. Gemini-Pro-1.5 and Qwen2.5-VL-72B-Instruct are called via OpenRouter; the results section also writes them as Gemini 1.5 Pro and Qwen2.5-VL-72B.

Experimental Setup

• System Under Test: The default backbone is GPT-4o, which also handles OCR verification, local candidate semantic scoring, RIS/Tavily checks, confidence evaluation, and report synthesis. Backbone replacements are Gemini-Pro-1.5 and Qwen2.5-VL-72B-Instruct. CLIP is only used for MemeDB encoding and is not reported separately. Temperature is 0 for all.
• Data: Meme100 has 100 items, including classic templates that have appeared on KnowYourMeme for over 10 years and new memes from the past year tied to high-impact events. Platforms are Reddit, Twitter/X, 4chan, Facebook, TikTok, YouTube, Instagram; posts with over 5,000 interactions were selected. Event examples include the 2024 U.S. presidential election, the Russia-Ukraine conflict, and large protests in Los Angeles in 2025. Appendix G provides statistics by topic, platform, modality (text-dominant, image-dominant, hybrid), and sparsity, but the main text does not give counts per category.
• Subset: The ablation subset has 20 items, with topic, platform, sparsity, and modality distributions aligned with the main set. The robustness subset applies three transformations to these 20 items: Gaussian blur (σ=2.5), border cropping (15%), and top UI overlay (about 12% coverage).
• Baselines: GPT-4o uses the same report prompt without external evidence. Appendix J additionally evaluates GPT-4o native web search. The paper does not numerically compare with classifiers like MOMENTA on Meme100.
• Human Scoring: Three in-team domain experts blindly evaluated. Each field was scored for Informativeness (whether information is complete) and Correctness (whether facts are correct) on a five-point Likert scale, averaged across annotators. adjusted_score is the mean of the two. The main experiment used Meme100; ablations and robustness used the corresponding subsets. Each annotator had 7,920 scoring items, totaling 23,760 across three. ICC was 0.502–0.600; Appendix I calls 0.502 moderate reliability.
• External Re-evaluation and Automatic Metrics: Appendix J involved two non-coauthor volunteers who blindly evaluated, with system identifiers removed and order randomized. Automatic metrics are computed from report text: numbers per 100 words, four-digit years, calendar dates, hedges, and platform mentions, plus spaCy entity counts on about/origin/spread. The paper does not use LLM judges and does not report repeated generation for the same image to estimate report variance.

Main Results

Swipe horizontally to read the table

System	Informativeness	Correctness	adjusted_score
GPT-4o VLM-only baseline	4.050	4.205	4.128
MemeTrace (backbone GPT-4o)	4.838	4.843	4.841

Per Table 2, Meme100, averaged over three annotators. The baseline and MemeTrace share the report prompt; the baseline has no external evidence.

• The authors report p < .001 for these three differences and claim an overall 17% improvement in adjusted score.
• In Table 1, MemeTrace scores higher than the baseline on both metrics for all 11 fields. The authors specifically highlight about, origin, and spread. safety_flag informativeness was 4.777 vs. 4.483, and correctness 4.670 vs. 4.400. The abstract claims more accurate detection of unsafe memes, especially disinformation and sensitive content; the main text does not provide separate detection accuracy or F1.
• Table 3: digits_per100w 2.088 vs. 0.631, years_per100w 0.880 vs. 0.168, dates_per100w 0.380 vs. 0.020, hedge_per100w 0.640 vs. 1.420, platforms_per100w 0.653 vs. 1.162, ner_person_unique 3.220 vs. 1.960. The authors state that the baseline, lacking evidence, more often lists multiple platforms, while MemeTrace mentions fewer platforms and ties them to evidence.

Module Ablations and Backbone Replacements

• Table 4 on the 20-item subset: full adjusted score 4.853; removing Tavily 4.227, removing RIS 4.317, removing LocalDB 4.388. The VLM baseline on this subset is 3.961. The authors state Tavily contributes most, RIS second, LocalDB third.
• Table 9 Informativeness: full about/origin/spread 4.967/4.733/4.700; removing Tavily 4.517/4.067/3.950; removing RIS spread 4.000. The authors state Tavily mainly supplements about and origin, and removing RIS weakens spread information.
• On the same 20 items, the Gemini version scored 4.451 and the Qwen version 4.575, both higher than 3.961 and lower than the GPT-4o version's 4.853. The authors believe n=20 cannot distinguish these two backbones, so they do not rank them.

Perturbations and External Blind Evaluation

• Table 4 adjusted scores: UI overlay 4.297, Gaussian blur 4.247, border cropping 4.229, compared to full 4.853. The authors state drops of about 0.56–0.62, roughly 12%; UI had the smallest drop. The authors believe blur mainly reduces correctness because text becomes unclear, hindering recognition; cropping reduces both because edge information is missing. In Table 4, blur informativeness/correctness is 4.292/4.202, and cropping 4.277/4.180.
• The authors state that in Table 9, about, origin, and structure are relatively stable. In that table's Informativeness, these three for the blur subset are 4.633, 4.367, 4.083, and for full 4.967, 4.917, 4.733. Origin drops more than about.
• In Appendix J, two external annotators on the same 20 items: MemeTrace adjusted score 4.790, GPT-4o native web search 4.594, retrieval-free baseline 3.964. MemeTrace was higher on 18 of 20 items. Combined ICC(C,k)=0.801. The authors state external evidence improves provenance quality, and the gain over general web search comes from specialized retrieval modules. These annotators are not the same as the three experts in Table 4.

Safety Flag

• Section 5.4 only analyzes safety_flag in Meme100 reports; human scores are in Table 1. The authors state disinformation appeared in 44% of reports and sensitive content in 16% (Figure 13a).
• The authors state political and geopolitical memes had the highest disinformation proportion, about 50%; technology-related memes were rarely flagged as sensitive (Figure 13b). In the co-occurrence matrix of Figure 13c, the authors state disinformation and fake misattribution co-occur strongly, and believe misinformation often accompanies incorrect source attribution. Specific cell values in the figure are not listed.
• Appendix L compares immigration/toll road, penguin tariffs, Last Supper parody, and UI-occluded variants. The authors state MemeTrace writes event background, while the baseline mostly restates visible text or imagery; under UI occlusion, some explanation is retained. The report text is not reproduced here.

Other Ablations and Analyses

• The strongest correlations between automatic metrics and human scores are Pearson r=0.78 and Spearman ρ=0.94; the authors state entity-type metrics align strongly, while hedge frequency and platform enumeration correlate negatively with human scores (Section 5.1, Appendix H, Figure 9). The main text does not assign 0.78 and 0.94 to a specific metric name.
• The main experiment averaged 18.46 steps, 11.54 multimodal LLM calls, and 3.92 external search API calls per meme (Tavily 2.92, SerpApi 1.00), plus about 3 local operations; cost $0.1259, end-to-end about 111 seconds (Appendix O). The authors state latency and API cost mainly come from iterative LLM reasoning.
• Figure 15 (n=100) has axes Iteration 1–5 and Confidence 0–1.00, with mean and ±1 std. The authors state average confidence increases with iteration and stabilizes around the fifth step, supporting the upper limit of 5. Endpoint values are not given.
• Appendix I: MemeTrace (full) overall score variance 0.1076, baseline 0.6025; the main text writes the main experiment overall variance as 0.108. Removing Tavily overall variance is 1.4739.
• Negative or exceptional cases per the authors: removing any of the three modules yields scores lower than full; both alternative backbones are lower than the GPT-4o version; all three perturbations are lower than the unperturbed subset; n=20 does not distinguish Gemini from Qwen.

Limitations and Future Directions Noted by the Authors

• MemeDB's scale does not yet cover the diversity of meme templates, especially new or non-English variants (Limitations).
• The system requires multiple LLM calls and online retrieval; the authors note the cost is high; analysis of efficiency, latency, and deployment cost is in Appendix O (Limitations).
• Current evaluations focus on natural perturbations; systematic red-teaming has not been done (Limitations).
• Potential vulnerabilities listed by the authors include source pollution from SEO or low-quality websites, and malicious injection or templated content in web pages. Domain scoring can mitigate some of this but cannot fully prevent information pollution (Limitations).
• Automatic report quality metrics are still imperfect. For example, the authors note that the frequency of dates or numbers only reflects information density, not factual accuracy (Limitations).

Experimental Coverage

• The main evaluation is Meme100, with 100 items; each generates a provenance report. Three in-team experts scored Informativeness and Correctness for 11 fields; two non-coauthors blindly evaluated three systems on a 20-item ablation subset (Section 4.3, Appendix J).
• Backbones are GPT-4o, Gemini-Pro-1.5, Qwen2.5-VL-72B-Instruct. Baselines are GPT-4o without external evidence, and GPT-4o native web search in Appendix J (Sections 4.2, 5.2).
• Module ablations disable Local Database Retrieval, RIS, and Tavily respectively, scored on the 20-item subset (Section 5.2, Tables 4 and 9).
• Robustness transformations are Gaussian blur (σ=2.5), border cropping (15%), and top UI overlay (about 12%), all based on the ablation subset (Sections 4.1, 5.3).
• The OCR module only extracts English (Section 3.3.1). The paper reports ICC, Pearson/Spearman correlations with human scores, and average call counts, cost, and latency in Appendix O; it does not report repeated generation of reports for the same image to estimate variance (Section 4.2 sets temperature to 0).

MemeTrace is a multimodal LLM agent that generates safety-related provenance reports from a single image.

• Problem: The authors argue that meme moderation cannot stop at safe/unsafe classification. Images depend on news, culture, and irony; static VLMs lack recent context and cannot explain implicit risks.
• Approach: Given an image, the system follows a Think–Act–Evaluate loop to extract English overlaid text, uses CLIP/FAISS to query MemeDB (21,305 confirmed entries, adopting only those with semantic scores above 8), and if insufficient, sequentially performs Google reverse image search and Tavily. A confidence evaluator decides whether to modify the query. The default maximum is 5 rounds, temperature is 0, and finally GPT-4o writes 11 fields, including a Safety Flag with fixed categories.
• Main Results: On Meme100, three experts gave an adjusted score of 4.841, and retrieval-free GPT-4o 4.128; the authors report p < .001 and claim an overall improvement of 17%. Safety flag correctness was 4.670 vs. 4.400. The authors state that disinformation appeared in 44% of reports.
• Ablations and Transfer: On the 20-item subset, full scored 4.853; removing Tavily dropped to 4.227, the largest drop among the three modules. The Gemini version scored 4.451, and the Qwen version 4.575, both higher than the subset baseline of 3.961 and lower than the GPT-4o version. The authors believe n=20 cannot rank these two backbones and attribute the gains to the agent design rather than any single VLM.
• Re-evaluation: Two external annotators gave an adjusted score of 4.790, higher than GPT-4o with native web search (4.594) and the retrieval-free baseline (3.964), with 18 of 20 items higher. The strongest correlations between automatic metrics and human scores were Pearson r=0.78 and Spearman ρ=0.94.
• Authors' Conclusions: External context is necessary for judging meme intent; reports can serve as research annotations and provide moderators with context that classifiers lack. The authors also note that MemeDB does not cover new templates and non-English variants, multi-round retrieval is costly, systematic red-teaming has not been done, and number frequency cannot replace fact-checking.