CyberSOCEval: A benchmark for evaluating LLM malware analysis and threat intelligence reasoning capabilities

Existing evaluations do not cover malware analysis and threat intelligence reasoning in SOC, leaving developers and users without benchmarks for selection and improvement.

• Scenario: The authors state SOC defenders are overwhelmed by alerts, threat intelligence, and changing business context, and need AI that enhances operational security work. The authors also state attackers are already using AI to scale cyberattacks, so open-source benchmarks are needed to drive improvement by defenders and model developers.
• Existing gaps: The authors state existing evaluations are incomplete for the scenarios most important to real defenders. The text names Sophos (2024) and Oniagbi (2024) as not open-sourcing their evaluations, and Bhusal et al. (2024)'s open-source benchmark as limited to the single industrial control industry.
• Authors' observation: Larger, newer LLMs tend to be better, which the authors state is consistent with training scaling laws. Reasoning models that exploit test-time scaling do not show the same gains seen in coding and math; the authors believe these models were not trained to reason about cybersecurity analysis.
• This paper proposes: An open-source suite, CyberSOCEval, part of CyberSecEval 4, containing Malware Analysis and Threat Intelligence Reasoning. The two tasks were selected by the team's security practitioners based on necessity in defensive operations. The authors state current LLMs are far from saturating the evaluation.

The authors position this paper as filling a gap in public SOC evaluations, focusing on defenders' reasoning in malware analysis and threat intelligence, rather than broad security knowledge Q&A.

SOC task evaluations

• Sophos (2024): Evaluates three SOC tasks, namely converting natural language questions into SIEM queries, summarizing events from raw SOC data, and rating event severity. The authors state the report emphasizes the need for public SOC benchmarks, but the evaluation was not open-sourced.
• Oniagbi (2024): Examines LLM agent assistance for Tier-1 SOC triage in a synthetic experimental environment. The authors relay its findings: assistance speeds up initial triage and reduces cognitive load, but works better as a copilot than as an autonomous agent; no open-source evaluation framework was provided.
• SECURE (Bhusal et al., 2024): Open-source evaluation of cyber defense skills in the industrial control domain. The authors state it covers only a single industry and does not meet the need for a more general benchmark suite.

Threat intelligence evaluations

• CTIBench (Alam et al., 2024): Four CTI evaluations, including threat knowledge multiple-choice questions, vulnerability-to-root-cause mapping, severity prediction per standards such as CVSS, and threat actor attribution from attack descriptions.
• SEvenLLM (Ji et al., 2024): An English-Chinese framework. Fine-tunes a domain LLM on 28 cybersecurity tasks using 8,485 threat reports, and evaluates CTI analysis capability with 1,300 Q&A samples.
• Authors' distinction: Relative to existing CTI evaluations, the authors state this paper adds multimodal reports (text IOCs with tables or figures) and sets high-level reasoning questions that current systems still find difficult but are human bottlenecks. The authors state that malware analysis from leaked reports was previously missing from evaluations, and this suite is novel in evaluating that capability.

Security knowledge and dual-use evaluations

• CyberSecEval 3 (Wan et al., 2024): Evaluates vulnerabilities such as insecure code and following malicious instructions, as well as dual-use capabilities like vulnerability identification. This paper's suite is part of the subsequent CyberSecEval 4.
• CyberMetric (Tihanyi et al., 2024): Generates thousands of multiple-choice questions from standards such as NIST and CWE, and conducts a broad security knowledge quiz across dozens of models.
• The authors state such work covers broad Q&A or specific technical subtasks, and state that general models often fall short on complex security reasoning.

Comparison in this paper's experiments

• The experiments do not use the above benchmarks as score baselines. The comparison is random guessing, computed in Appendix B (Malware Analysis) and Appendix C (Threat Intelligence Reasoning) respectively.

The authors state CyberSOCEval tests defender reasoning in SOC contexts more deeply, requiring context integration, multi-step reasoning, and judgment. The conclusion states that, to their knowledge, this is the first open-source LLM benchmark focused on representative SOC activities.

CyberSOCEval turns malware detonation logs and threat intelligence reports into automatically gradable multiple-choice questions, using LLM-generated questions with human revision to control question quality.

Question Selection Criteria and Scoring

• Three design criteria: high performance should translate into real efficiency gains; evaluation should be fully automatic; question volume should be large enough to reliably detect moderate performance differences. The experiments themselves did not measure SOC work hours.
• Question type and accuracy: both are multiple-choice with potentially multiple correct answers. Accuracy is the proportion of questions where all correct options are selected and no incorrect options are selected. The appendix additionally uses Jaccard to give partial credit for selections that partially overlap with the correct set.
• Quality control: iterative LLM generation, human review, per-question revision, and pipeline updates. The introduction states that the appendix describes the human review protocol and bootstrap-based estimation of statistical power and minimum detectable effect; the main text does not give numerical values for these effects or quality thresholds.

Malware Analysis

• Data: publicly available malware targeting Windows enterprises is detonated in a controlled environment, using Hybrid Analysis reports produced by CrowdStrike Falcon Sandbox. Logs are JSON, containing processes, extracted files, static signature detections, etc.
• Categories and scale: Table 1 has five categories: EDR/AV KILLERS, RANSOMWARE, RAT-REMCOS Family, INFOSTEALERS, UM UNHOOKING. There are 609 Q&A pairs in total.
• Question generation: Llama 3.2 90B synthesizes interview-style Q&A from reports, and cybersecurity experts validate and revise each question. The main text states up to ten possibly correct options; the random baseline in Appendix B is computed with 9 options and number of correct options K=1 to 9.
• Pipeline: Figure 2 shows the generation and revision steps; Figure 3 is a shortened illustrative question, not the full sandbox JSON. The authors believe capabilities on malicious file logs may also transfer to non-file-triggered logs, and state this awaits validation with more log sources.

Threat Intelligence Reasoning

• Input and scoring: each report is given as images, one page per image. Up to six options, multiple selection allowed. Accuracy is defined the same as in the malware benchmark. Figure 6 is an illustrative pipeline.
• Two synthesis strategies: using Llama 4 Maverick and Llama 3.2 90B. The category strategy uses a manually defined list of static categories, has the LLM generate questions from extracted text, then validates for clarity, correctness, and relevance.
• Relationship strategy: the LLM turns report text into a graph-like representation of key relationships, with examples including threat actors, malware, and techniques, then samples and manually validates until reaching what the authors describe as a quality threshold that makes test-item accuracy statistically significant. The paper does not give this threshold.
• Scale: there are also a small number of manually created questions whose answers rely entirely on figures. The final set has 588 questions from 45 reports, with types including APT activity analysis, malware technical reports, and vulnerability advisories. In Appendix C, with number of correct options from 1 to 6, question counts are 201, 256, 113, 12, 2, 4 respectively. Figure 7 plots question and report counts by source; the main text does not list per-source counts.

Run Settings and Log Filtering

• Systems under test: recent models from OpenAI, Google, Anthropic, Meta, Deepseek, with original parameters and no fine-tuning. temperature, top_p, and thinking level remain at provider defaults. When many answers were unparseable, system prompts were adjusted per model to conform to the format; the paper does not provide these prompts. Scoring is exact matching against the reference answer, not an LLM judge.
• Short context: Hybrid Analysis reports often approach or exceed some models' 128,000-token window. The optional filter keeps only fields listed by the authors, truncates signature descriptions to 50 characters, and replaces 32-character hashes with shorter alphabetic hashes. Appendix D lists the retained fields. The authors state in Section 4.1 that abbreviated reports generally do not materially change scores.

The authors report two overall accuracies on recent, non-fine-tuned models: malware analysis approximately 23-34%, and threat intelligence reasoning approximately 43%-53%, and state both are well above random guessing and far from saturation. Per-model scores are in figures; the text does not list them cell by cell.

Experimental setup

• Tested models: Recent models from OpenAI, Google, Anthropic, Meta, and DeepSeek, with default temperature, top_p, and thinking level, not fine-tuned. The text names o3, deepseek-r1, 4o, and gemini-2.5-pro; the Figure 11 discussion states six tested models in total, with the remaining names not listed in the text.
• Malware questions: 609 questions across five categories per Table 1, from Hybrid Analysis reports of public Windows enterprise-targeted samples in a controlled environment. The acknowledgments state the benchmark code had an error, was corrected, and results were updated in October 2025. Appendix B's single-option baseline uses 236/608, inconsistent with the 609 in the main text; the paper does not explain this.
• Threat intelligence questions: 588 questions from 45 reports. The main results correspond to inputting reports as multiple images. Appendix C computes the random baseline with 6 options.
• Metrics: Accuracy requires selecting all and only the correct options. The appendix reports Jaccard. Figure captions give 95% confidence intervals by normal approximation, and state avg n for parseable answers with row totals; the text does not give parseable counts per model.
• Question-generation models: Llama 3.2 90B; threat intelligence also uses Llama 4 Maverick. Experts revised manually. The paper does not report repetition counts, human agreement, or quality threshold values.

Main results

The table below only includes overall ranges and random baselines stated in the text. Breakdowns by difficulty, malware family, question topic, or report source are in Figure 8, Figure 9, and Appendix A; the text does not give per-item numbers.

Benchmark	Accuracy	Random baseline
Malware Analysis	approximately 23-34% (Section 4.1, corresponding to Figure 8)	approximately 0.63% (Section 4.1; Appendix B)
Threat Intelligence Reasoning	approximately 43%-53% (Section 4.2, corresponding to Figure 9)	approximately 1.7% (Section 4.2; Appendix C gives 1.739%)

• Scaling: The authors state larger, newer LLMs tend to be better, consistent with training scaling laws. The text has no table sorted by parameter count.
• Not saturated: The authors state no current LLM saturates the evaluation, so models still need tuning for cyber defense.
• Selection: The authors state results differ across models, and the evaluation can help practitioners choose LLMs. The paper does not give variance values.

Multimodal input

• Setup: Section 5.2 and Figure 10 compare accuracy and mean Jaccard on the same threat intelligence benchmark: extracted text, images only, and text plus images. The main results in Section 4.2 correspond to image input.
• Results: The authors state that on all tested models, extracted text outperforms images only; text plus images is close to text only, sometimes slightly worse. Figure 11 shows one initial-access-vector question: all six tested models answer correctly with image input; with text only, only o3 answers correctly; with both inputs, only o3, 4o, and gemini-2.5-pro answer correctly.
• Authors' interpretation: The authors believe that on this benchmark, vision models perform better with text than images, indicating room for improvement in reading text through other modalities; when both modalities are given, models may over-rely on text. The authors also state the dataset is text-biased, with a small number of purely visual handcrafted questions.

Test-time thinking

• Comparison: Section 5.3 compares models that exploit reasoning-time thinking (examples: o3, deepseek-r1) with other models.
• Results: The authors state they did not observe the larger effects seen in coding and math, and state they were initially surprised. The paper does not report percentage-point differences for this comparison.
• Authors' hypothesis: Test-time thinking may matter for defensive results, but may also require targeted training with security-related data and reasoning traces. The authors see this as an opportunity to improve foundational reasoning in the security domain.

Other ablations and analyses

• Hybrid Analysis filtering (Appendix D, Figure 15): For most models, ceiling metrics on full text are a few percentage points higher than on truncated versions; the authors state no statistically significant difference was observed and believe models mainly use a subset of telemetry.
• Appendix A, Figures 12-14 give overall accuracy, Jaccard, and breakdowns by malware family and question topic; the text does not give per-item numbers.
• Another random strategy: guessing only 1 option in malware gives expected accuracy of approximately 4.3% (Appendix B); guessing 2 options each time in threat intelligence gives approximately 2.902% (Appendix C).
• Section 5.1: The authors believe file-execution-triggered logs also carry signal for fileless attacks, because the log footprint of most system events can be identical; they also state that IP exfiltration, access to sensitive assets, and similar may not be covered.

The limitations the authors write are that multiple-choice questions and benchmark distributions cannot directly represent production tasks, and they plan to expand attack types, runtime environments, non-text modalities, and triage evaluation.

Limitations and Future Directions Noted by the Authors

• Multiple-choice questions: cannot perfectly proxy free-form answering, nor deciding when to act on one's own. The authors accept reduced realism for reproducibility and interpretability, and use more options and multiple correct answers to lower guessing probability (Section 5.4).
• Synthesis bias: if the model under test is the same as or similar to the Llama 3 or Llama 4 used for question generation, apparent scores may be inflated. The authors state that after a multi-agent pipeline and extensive human revision, this remains a potential bias (Section 5.4).
• Not literally extrapolable: the benchmark context, and the distribution of attacks or related intelligence, may differ from production data at a particular enterprise or SOC. The authors state that accuracy cannot literally predict production metrics and only provides a relative signal for security decisions; organizations may still need internal test sets and further fine-tuning (Section 5.4).
• Attacks and environments: the authors plan to add fileless attack telemetry and sensitive data exfiltration, and extend the current Windows system logs to Linux and mobile environments (Section 5.1, Section 5.5).
• Modalities and ontologies: the authors plan to add questions that require jointly interpreting text and non-text inputs. They hope to do entity extraction and relationship mapping, and state that two obstacles are ontology misalignment and subjectivity in mapping ground truth; apart from MITRE ATT&CK, there is no universally accepted framework of security concepts (Section 5.5).
• Triage and new models: the authors hope to add incident response and triage evaluation, with the difficulty being the subjectivity of "correct" prioritization and business contexts that are both comprehensive and realistic. The authors also want to see the performance of Claude 4, GPT-5, and Foundation-Sec-8B, released after the research was completed, on this benchmark (Section 5.5).

Experimental Coverage

• Two completed tasks: Malware Analysis has 609 questions across five categories shown in Table 1, with logs from CrowdStrike Falcon Sandbox / Hybrid Analysis of Windows samples; Threat Intelligence Reasoning has 588 questions from 45 reports (Section 3).
• Systems under test are recent models from OpenAI, Google, Anthropic, Meta, Deepseek, without fine-tuning, with temperature, top_p, and thinking level at provider defaults. The main text names o3, deepseek-r1, 4o, and gemini-2.5-pro; Figure 11 discusses six models under test (Section 4, 5.2).
• The primary metric is accuracy requiring all correct selections, with Jaccard additionally reported in the appendix; random baselines are in Appendices B and C. Figure captions show 95% confidence intervals based on normal approximation.
• Also compared are report images, extracted text, and both together (Figure 10), and full Hybrid Analysis text versus field filtering (Figure 15, Appendix D).
• The paper does not report number of repetitions, human agreement, or quality threshold values, nor does it list per-model accuracy in the main text. Question generation is stated to use Llama 3.2 90B and Llama 4 Maverick, with the pipeline limited to Llama 3 and Llama 4.

The authors propose the open-source benchmark suite CyberSOCEval, as part of CyberSecEval 4, to measure how well LLMs support SOC-related human work. The authors state that, to their knowledge, this is the first open-source LLM benchmark focused on representative SOC activities.

• Problem: existing evaluations have incomplete coverage of malware analysis and threat intelligence reasoning; some SOC research is not open source, and open-source benchmarks for industrial control systems do not cover more general scenarios.
• Approach: both tasks are multiple-choice, requiring selection of all and only the correct options. The malware side uses sandbox JSON, 609 questions, generated by Llama 3.2 90B and then revised by experts. The threat intelligence side converts 45 reports page by page into images, 588 questions, generated by Llama 4 Maverick and Llama 3.2 90B using category or relationship strategies, with a small number of purely figure-based manual questions.
• Results: on recent models without fine-tuning and with default hyperparameters, the authors state malware analysis accuracy is approximately 23-34%, against a random baseline of about 0.63%; threat intelligence reasoning is approximately 43%-53%, against about 1.7%. Per-model scores are in Figures 8 and 9.
• Other findings: the authors state that larger, newer models tend to do better; reasoning models such as o3 and deepseek-r1 do not show the improvements seen in coding and math. Extracted text outperforms images alone. Between full text and filtered logs, the authors state they see no statistically significant difference.
• Conclusion: the authors state the evaluation is far from saturated and can provide improvement directions for model development and allow practitioners to compare models. The authors also state that scores cannot literally predict production metrics for a particular SOC, and organizations may still need internal test sets and further fine-tuning.