BypassGym: a benchmark for evaluating AI agents' endpoint defense evasion capability
BypassGym: A Benchmark for Evaluating AI Agents' Endpoint Defense Evasion Capabilities
The authors propose the BypassGym benchmark to measure AI agents' ability to generate loaders that bypass real EDR; across 90 runs with 5 model backends, 71 succeeded.
- Existing agent cybersecurity benchmarks test penetration testing and vulnerability discovery, but not defense evasion: whether an agent can write a loader that executes a payload on an endpoint protected by a real EDR without being blocked. The authors argue that LLMs have already been used for malware development in real attacks, making this question urgent.
- Proposes BypassGym: given fixed C2 shellcode, an agent iteratively develops a loader inside a container using a compilation toolchain, first verifying functionality on a VM without EDR, then performing execution and C2 functionality checks on a VM with the target EDR installed, with structured feedback for the next round of modification; it comes with an evasion knowledge skill of adjustable richness.
- 5 model backends against 6 EDRs and 3 payloads, 90 runs in total, 71 (78.9%) succeeded; DeepSeek-V4.1 Flash reached 17/18, Bitdefender allowed only 5/15. Providing evasion knowledge improves the success rate; detailed endpoint feedback helps, while pass/fail-only feedback does not.
- The authors note: only 6 commercial EDRs on Windows are covered; knowledge base coverage is limited, so success rates should be viewed as a lower bound on capability; detection of the loader versus the payload cannot be fully distinguished. RQ2/RQ3 use only two models, GLM and DeepSeek, with 18 runs per condition and one run per combination.
- Threat model
- Task setting: given a fixed C2 shellcode payload, the agent must generate and compile a loader that executes the payload on a real Windows endpoint with the target EDR installed, without the process being blocked or terminated; judgment excludes detection alerts and looks only at functional execution and survival. The agent may use the provided evasion knowledge skill and the Windows compilation toolchain, with at most 10 iterations per run.
- Evaluated models
- GPT-6 LunaGemini 3.1 Flash LiteGLM-5.3 FlashDeepSeek-V4.1 FlashQwen3.7 Plus
- Benchmarks
- BypassGym(基于 Proxmox VE 的 Windows 端点环境,6 个 EDR 产品 × 3 个 C2 payload:Havoc、AdaptixC2、Sliver)
- Metrics
- 运行成功率(passing submission 数/运行数)迭代次数、耗时、token 消耗延迟重放存活率
Researchers release the BypassGym benchmark, which evaluates AI agents' ability to bypass EDR (Endpoint Detection and Response) on real endpoints; the authors say this is the first benchmark to systematically evaluate this capability on real endpoints. The task is set as follows: given C2 shellcode, the agent must generate and compile a loader so that the payload executes without being blocked or terminated by the deployed EDR; the benchmark also provides supplementary evasion knowledge and skills, supporting comparison with and without this input. The environment is built on Proxmox VE, runs real Windows systems and EDR products in virtual machines, and returns structured detection feedback to the agent. The evaluation covers 5 lightweight model backends, 6 EDR products and 3 payloads; results show that even these simple models can generate submissions that successfully bypass, with DeepSeek-V4.1 Flash reaching 17/18 successful runs, and Bitdefender allowing only 5 of 15 runs.
Paper analysis
What problem does this paper attempt to solve?
Existing benchmarks do not test defense evasion: whether an agent can write a loader that executes a payload on a real EDR endpoint.
Core question: can AI agents autonomously generate loaders that execute C2 payloads on Windows endpoints protected by real EDR without being blocked or terminated? Existing benchmarks do not measure this capability.
- Scenario and motivation: the authors state that LLMs have already been used in real malware workflows (cases such as LAMEHUG and PROMPTFLUX), and that EDR is a key line of defense, yet whether agents can bypass it is unclear; model providers, endpoint security vendors, and policymakers all need an answer.
- Shortcomings of existing evaluations: the authors state that existing AI malware research evaluates samples via scanning services, sandboxes, or offline detectors, and that agentic security benchmarks mainly test penetration testing, vulnerability discovery, and reproduction; none systematically measure, on real endpoints with deployed EDR, whether a loader can make a payload actually run.
- Three challenges: standardize the task without constraining loader design; evaluate on real endpoints and restore a clean state between tests; domain knowledge must be a controllable input so its effect can be measured.
- The paper proposes: the BypassGym benchmark—fixed payload, tools, and iteration budget, open loader implementation, running real Windows systems and EDR products on Proxmox VE, returning structured detection feedback, and accompanied by an evasion knowledge skill of adjustable richness.
- Task setup (threat model):
- Goal: generate and compile a loader so that the given C2 shellcode executes on the target endpoint, the process survives, and a usable C2 session is established.
- Inputs: fixed payload, knowledge skill (three levels: full/partial/none), README, and a Windows compilation toolchain.
- Victim system: a Windows endpoint with a real EDR product installed in a Proxmox VE virtual machine.
- Judgment: based on functional execution and survival; detection alerts are not counted in the judgment (the paper explains this is because the products have no consistent public alert interface).
What related research exists?
Related work falls into two categories: AI malware generation and evasion, and agentic cybersecurity benchmarks; the authors state that none of them test execution on real EDR endpoints.
AI malware generation and evasion
- MalGEN [31]: uses planner and developer agents to generate malware according to MITRE-specified behaviors, evaluated via VirusTotal and a sandbox.
- LLMalMorph [2]: mutates Windows malware source code into hundreds of variants, testing evasion and functionality preservation, evaluated via VirusTotal and Hybrid Analysis. The paper notes that both are evaluated via aggregate scanning, sandboxes, or offline classifiers, and do not test whether the payload runs on the endpoint.
- Dante-7B [29] and Obfusc8 [18]: the former fine-tunes a small model to generate compilable shellcode loaders, and the paper notes it does not test execution against EDR and does not support agent iteration; the latter tests deployed endpoint defenses but only rewrites PowerShell scripts.
Cybersecurity evaluation of LLMs and agents
- Cybench [38] and AutoPenBench [11]: evaluate CTF solving and penetration testing, respectively, judged by task success.
- CyberGym [36]: vulnerability discovery and reproduction, with sanitizer crash as the oracle. The authors state it is closest to BypassGym (both use execution-based oracles in isolated environments), but the artifacts and results evaluated differ.
- AgentCyberRange [20] and CyberSecEval 3 [35]: the former tests web exploitation and post-exploitation in a real network range, where EDR evasion is only one step in the attack chain; the latter includes an autonomous evasion phase, but evaluates command output with an LLM judge, and the paper notes it does not test whether generated samples evade detection while preserving malicious functionality.
Baselines and comparison setup
- The paper has no method baseline; the functional baseline is the default EXE generated directly by the three C2 frameworks (no agent, no custom loader), which failed 15 and passed 3 out of 18 EDR cells (Table 3).
Authors' positioning: the authors state that BypassGym is, to their knowledge, the first benchmark to systematically evaluate defense evasion capability on real endpoints, using "the payload actually executing under EDR" as the evaluation oracle (Table 1).
How does the paper address this problem?
BypassGym three modules: task setting, in-container loader iterative development, EDR evaluation and structured feedback on PVE virtual machines.
Overall approach: define "fixed C2 shellcode + target EDR" as a task; the agent iteratively develops a loader inside a container, first verifying functionality on a VM without EDR, then performing execution and C2 functionality checks on a VM with the target EDR installed, with feedback for the next round of modification. BypassGym contains three modules (Figure 1).
Evasion Task Setup
- Task inputs: fixed C2 shellcode payload (the current instance uses Havoc, AdaptixC2, Sliver), the skill under the selected knowledge condition, and a README describing the deliverable; Appendix A gives a representative initial prompt.
- Knowledge base: 37 independent techniques, 3 complete reference projects (AceLdr, gargoyle, BOAZ), and 6 DLL hijacking templates, organized by loader stage and packaged as a native skill following the OpenAI Codex skills documentation.
- Three knowledge conditions: full provides the complete skill; partial removes all README.md but keeps source code, binaries, and loader projects; none gives only SKILL.md containing metadata.
- Deliverable constraints: the ZIP contains exactly one .exe entry point (PE32+/x86-64), may include dependencies and a separate loader DLL; the shellcode within the bundle must be stored encrypted.
Shellcode Loader Development
- Runtime environment: the agent scaffold and model backend are configurable inside the container; the experiments use Codex CLI, Claude Code, OpenCode and the GPT, Gemini, GLM, DeepSeek, Qwen backends; the workspace persists across iterations.
- Runs and iteration: a run fixes the payload and target EDR, with at most 10 iterations; a candidate must first pass the Endpoint Execution Check on the NoEDR VM before entering the target EDR VM; when NoEDR fails, up to 10 fix submissions are allowed within the same round.
- Success criterion: the first submission that passes all required checks on the target EDR is a success; exhausting 10 iterations is recorded as failure.
EDR Evaluation and Feedback
- Infrastructure: Proxmox VE virtual machines, including one NoEDR VM (Windows 11, Defender removed) and one Target EDR VM per product; snapshots are restored before each test.
- Evaluation oracle: based on functional execution and survival—the payload actually executes and the process is not blocked or terminated within the observation window; detection alerts are deliberately excluded, which the authors explain is because the products have no consistent public alert interface and real attackers may not care whether an alert is triggered.
- Two sets of checks (Table 2): the Endpoint Execution Check runs four stages on the endpoint VM (Static Check, Process Check waiting 10 seconds after launch to check survival, Traffic Monitor observing for 60 seconds requiring an outbound connection, Delayed Check waiting another 10 seconds); the C2 Functionality Check runs on the C2 server, using a unified playbook to verify command execution and bidirectional file transfer.
- Feedback: both stages return structured results, on which the agent bases its modifications to the submission in the next iteration.
What experiments did the paper conduct?
5 models × 6 EDRs × 3 payloads, 90 runs in total, 71 (78.9%) succeeded; knowledge, scaffold, reasoning effort, and feedback granularity were also tested.
Experimental Setup
- Models tested: five backends: GPT-6 Luna, Gemini 3.1 Flash Lite, GLM-5.3 Flash, DeepSeek-V4.1 Flash, Qwen3.7 Plus; RQ1 all use Codex CLI.
- Environment: isolated Windows virtual machines on Proxmox VE; 6 EDR products (Windows Defender, Kaspersky, Elastic, Trend Micro, Sophos, Bitdefender); payloads generated by AdaptixC2, Havoc, Sliver; versions in Appendix C.
- Run scale: RQ1 is 5×6×3=90 runs, one run per combination; RQ2/RQ3 use only GLM and DeepSeek (the two with the highest success rates in RQ1), 6×3=18 runs per condition.
- Metrics: run success rate (passing submissions/18 or /90), iteration count, time, token consumption; delayed replay survival rate.
- Judgment method: the two sets of checks in Table 2 (four stages of endpoint execution + three C2 functions), no LLM judge.
- Functional baseline: default EXEs from the three frameworks submitted directly, 15 failed and 3 passed out of 18 EDR cells (Table 3).
Main Results
RQ1 overall success rate by model (per Table 4, Table 5, 18 runs per model):
Model backend Successful runs Runs requiring NoEDR fixes DeepSeek-V4.1 Flash 17/18 (94.4%) 3/18 (16.7%) Gemini 3.1 Flash Lite 15/18 (83.3%) 8/18 (44.4%) GLM-5.3 Flash 15/18 (83.3%) 13/18 (72.2%) GPT-6 Luna 14/18 (77.8%) 5/18 (27.8%) Qwen3.7 Plus 10/18 (55.6%) 10/18 (55.6%) - Overall: 71 of 90 runs (78.9%) generated passing submissions; by EDR, Windows Defender passed all 15 submissions, Bitdefender was the hardest, passing only 5/15 (33.3%); by payload, Sliver 27/30, Havoc 24/30, AdaptixC2 20/30.
- Authors' interpretation: the authors state that all models produced passing submissions; 50 of the 71 passing submissions (70%) used DLL side-loading via the signed host ssvagent.exe; all 5 models failed to pass Havoc on Bitdefender, which the authors describe as a common difficulty under this test setting.
- Execution mode: 58/71 successful runs required only one Target EDR test; fix overhead varied widely (DeepSeek 3 NoEDR fix calls, Gemini 141, GLM 212); 4 runs had refusals totaling 135 turns, of which 97 turns were concentrated in one failed GPT run.
Delayed Replay (RQ1 addendum)
- What was tested: the 71 passing submissions were replayed on the original target EDR 1–4 weeks after the original run, waiting 24 hours after snapshot restoration for EDR to update detection libraries.
- Results: 62/71 (87.3%) still passed all four stages (Table 6); Trend Micro, Sophos, Bitdefender had no losses, Windows Defender lost 1, Kaspersky 5 of 13 replays no longer passed (survival rate 61.5%), Elastic 76.9%; by payload, all replayed Havoc submissions survived, 5 of 20 AdaptixC2 replays failed.
- Authors' interpretation: the authors state that most original successes remained undetected in delayed replay, with clear differences among defenses.
Knowledge, scaffold, and reasoning effort (RQ2)
- Knowledge richness (Table 9): DeepSeek success rate dropped from 17/18 with full to 13/18 with none; GLM from 15/18 to 9/18. The authors state that external evasion knowledge provides usable methods and reusable code, and GLM relies on it more; cost does not change monotonically with knowledge richness.
- Scaffold (Table 7, Table 10): GLM was 15/18 under all three scaffolds; DeepSeek was 17/18 under Codex CLI and Claude Code, 16/18 under OpenCode. Claude Code reduced GLM's NoEDR fixes from 212 to 34, but raised DeepSeek's Target EDR tests from 32 to 46; the authors state that scaffold effects depend on the model.
- Reasoning effort (Table 11): GLM was 11/18, 14/18, 14/18 under Low/Medium/High; DeepSeek was 17/18, 16/18, 16/18. The authors state that increasing reasoning effort does not reliably improve success rate, and DeepSeek was already best at low effort.
Endpoint feedback (RQ3)
- What was tested: comparing three feedback types: one-shot (only one target submission), binary (only know success or failure), detailed (structured judgment, including failed check items).
- Results (Table 8): GLM was 14/18, 14/18, 15/18 under the three settings; DeepSeek was 14/18, 14/18, 17/18.
- Authors' interpretation: the authors state that pass/fail-only feedback does not improve overall success rate, while reporting which checks failed can improve success rate; the case in Figure 3 shows DeepSeek passing after modifying the loader based on Sophos failure feedback, but the authors note that a single case cannot determine which modification caused the pass.
Other ablations and analyses
- Functional regression: in 17 runs, a candidate passed NoEDR and then failed on Target EDR, and the next submission no longer passed NoEDR, for 26 such transitions in total.
- Bitdefender-specific failure mode: in some failed tests, Bitdefender traced to the evaluation service's Python Guest Agent and deleted the script, and the test was judged failed by timeout and heartbeat loss; the authors state this result is never recorded as a pass.
- Refusals: in one GLM run, it refused 35 times and still passed the Target EDR test.
What are some points that can be further explored?
The authors list three limitations: coverage, knowledge base lower bound, and loader/payload attribution, and propose directions for extension.
Limitations and future directions noted by the authors
- Coverage (Limitations and future work): currently only 6 commercial EDR products on Windows are tested; the authors state that extending the tasks to other operating systems, and extending to evasion of the shellcode payload itself (rather than only the loader), are natural next steps.
- Knowledge base coverage (same section): the knowledge skill has limited coverage and focuses on loaders; the absolute success rate should be regarded as a lower bound on agent capability under the given knowledge conditions.
- Attribution issue (same section): the current design cannot fully distinguish detection of the loader from detection of the payload; differences between payloads should not be interpreted as the agent's ability to evade detection through loader design alone.
- Future direction one (same section): expand the knowledge skill into a composition graph of evasion techniques, recording which techniques can be combined and which conflict, and empirically validate it through agent generation and testing.
- Future direction two (same section): how to automatically and precisely identify why a submission was detected.
- Reproducibility (Reproducibility section): the authors plan to release the benchmark framework source code and complete experimental data after review; licensed EDR installations, credentials, original C2 shellcode, or directly executable submissions cannot be redistributed.
Experimental coverage
- 5 model backends were tested in total; RQ2 and RQ3 used only two models, GLM-5.3 Flash and DeepSeek-V4.1 Flash.
- Each model × EDR × payload combination was run only once; RQ1 had 90 runs in total, and RQ2/RQ3 had 18 runs per condition.
- All RQ1 runs used Codex CLI; the scaffold comparison covered three: Codex CLI, Claude Code, and OpenCode.
- The evaluation oracle only looks at functional execution and survival; detection alerts are not counted in the judgment; the paper does not report alert telemetry for each EDR.
- Delayed replay covered all 71 passing submissions, conducted 1–4 weeks after the original runs (2026-09-15 to 2026-09-29).
Summarize the main content of the paper
BypassGym tests agent evasion capability on real EDR endpoints: 71 of 90 runs succeeded, and knowledge and detailed feedback helped.
BypassGym is a benchmark for evaluating AI agents' endpoint defense evasion capability, which the authors state is, to their knowledge, the first benchmark to systematically measure this capability on real endpoints.
- Problem: existing benchmarks test penetration testing and vulnerability discovery, but do not test whether agents can write loaders that actually execute C2 payloads under deployed EDR; the authors state that LLMs have already been used for malware development in real attacks, and this capability urgently needs measurement.
- Method: given fixed shellcode (Havoc, AdaptixC2, Sliver), the agent iteratively develops a loader inside a container, first verifying functionality on a VM without EDR, then passing the four-stage endpoint execution check and the C2 functionality check on a Windows VM with the target EDR installed; accompanied by a knowledge skill of 37 techniques and 3 reference projects, in three levels: full/partial/none.
- Main results: 5 model backends × 6 EDRs × 3 payloads, 90 runs in total, 71 (78.9%) succeeded; DeepSeek-V4.1 Flash reached 17/18, Qwen3.7 Plus lowest at 10/18; Windows Defender allowed all 15 submissions, Bitdefender only 5/15.
- Ablation: after removing knowledge, DeepSeek dropped from 17/18 to 13/18 and GLM from 15/18 to 9/18; the scaffold's effect depends on the model; increasing reasoning effort does not reliably improve the success rate; detailed feedback improves the success rate more than pass/fail-only feedback.
- Delayed replay: 71 passing submissions replayed 1–4 weeks later, 62 (87.3%) still passed; Kaspersky had the lowest survival rate (61.5%).
- Authors' conclusion: the authors state that low-cost defense evasion is already effective in practice and recommend that defenders correlate loader compositions with telemetry across layers; the success rate should be regarded as a lower bound on capability, and the benchmark will be extended to other operating systems and payload-level evasion.