Cyber range evaluation of autonomous cyber incident response agents

The core question is: can a reinforcement learning (RL) defensive agent trained in simulation, when deployed to a network range designed for human training, block red team expansion of access at lower combined cost than a hand-crafted rule-based strategy?

• Scenario and importance: The authors state that cyber attacks and security incidents occur frequently, and rapid mitigation is needed to prevent sensitive information theft or critical system unavailability; mitigation is usually done manually by network operations staff, or automatically by pattern- and rule-based systems.

• Limitations of existing methods: The authors state that existing work on RL for network intrusion response, with few exceptions, is only evaluated in simulation and not on the simulated target system; simulations assume system dynamics and agent-available data. The authors also cite Wolk et al. and Hicks et al., stating that simulation-trained network defense agents tend to overfit to specific topologies or attacker strategies.

• Core hypothesis: The hypothesis the authors test is whether RL-based agents can complete network defense tasks in a networked computer environment more effectively than hand-crafted rule-based agents. Effectiveness is measured by combined cost: blocking red team access to hosts while reducing the availability cost of defensive measures.

• What this paper does: The authors evaluate heuristic strategies and RL-learned strategies on ADS-24, a 37-virtual-machine network in the Crate range. Strategies are optimized using the MAL Simulator on a system model built with the Meta Attack Language (MAL), then connected to the target network.

• Task boundaries (not written by the authors as an attacker threat model):

  • Goal: The defensive agent blocks red team tool Lore from expanding access within a fixed two-hour window and minimizes operational cost.
  • Knowledge and input: The agent does not directly see network state, only partially observable facts mapped from Wazuh events to MAL; the authors do not call this setting white-box or black-box.
  • Capabilities: Each step, choose one action and one target object; actions are wait, shutdown (Application.notPresent), or bidirectional subnet traffic blocking (ConnectionRule.restricted).
  • Victim system: ADS-24 in the Crate event plane; Lore enters from flightlogs, which has no Wazuh agent, and the authors state it therefore cannot be fully expelled from the network.

The authors place the related work in the machine learning lineage of autonomous cyber defense and position their design choices along four axes: modeling, simulation training, architecture, and evaluation.

Task and Formalization

• Hicks et al.: They split cyber defense modeling into defense task modeling and system modeling; the authors adopt this decomposition.
• POMDP and cost: Several prior works formulate defense as a partially observable Markov decision process, with rewards encoding operational requirements; the authors follow the setting of "maintaining operations and minimizing the cost of security violations" and assume the process is factored and relational.
• Provenance graphs: The authors mention that kernel-level system call-based providence graphs have been used for intrusion detection and reinforcement learning; the authors note they are fine-grained, grow quickly, and that attackers may directly affect the defensive agent's observations.

Simulation Training and Sim-to-Real

• CybORG/CAGE, Yawning Titan, CyberWheel: The authors state that sim-to-real in autonomous cyber defense has given rise to these simulation environments; CAGE 2 is used by the authors as an example of singleton testing, Cage 4 has scenario variation but is tied to specific infrastructure, and Yawning Titan is more flexible in configuration but higher in abstraction.
• Kirk et al.: They distinguish singleton, in-distribution, and out-of-distribution testing; the authors use this taxonomy to explain why a simulator with configurable topologies and attacker strategies is needed.
• Hicks et al. and surveys: Hicks et al. outline the challenges of building cyber defense simulation environments, including accurately modeling infrastructure and mission elements; Vyas, Mavroudis, and Burnap, as well as Drašar et al., survey the pros and cons of various simulators.

Policy Architecture and Evaluation Environment

• MLP and GNN: The authors state that related work often uses multilayer perceptrons on flat vectors, which locks the policy to a specific state size and ordering; graph neural networks can encode topology, scale linearly, and have been used in autonomous cyber defense. The authors choose Vejde for the GNN implementation.
• CSLE, CyberWheel, GNS3, Crate: Hammar's CSLE is tested in a container range and uses range data for simulation training; CyberWheel provides both virtual machine and simulation interfaces; Le et al. use GNS3. The authors state that containers are lightweight but sometimes cannot represent the security properties of real networks. This evaluation uses Crate, a range designed for human training.
• Attack and user simulation: Hammar uses a pre-defined action list, CyberWheel uses heuristics to select actions; the authors state that Lore or CALDERA can choose among thousands of actions per step. Benign activity is commonly simulated with scripting tools such as GHOSTS.

Baselines and benchmark: The comparison baselines are Heuristic (selecting the corresponding defense step when a related attack step is observed) and NoOp (taking no action) on the same MAL data model; the evaluation environment is ADS-24 in Crate, and the training environment is the MAL Simulator.

The authors state that this work differs from work evaluated only in simulation: they build the ADS-24 model with MAL, train in simulation, then evaluate against Lore in the range, and check whether the policy is robust to problem variations.

The overall pipeline is sim-to-real: first, a system model is built using a modeling language and asset inventory, defense strategies are optimized in a simulator, and then the strategies are connected to the target network with red team and simulated users. Events are translated through an interface into the data model, and actions are translated back into host commands.

Network and Task Model

• Environment: ADS-24 is a fictional maintenance management system, 37 virtual machines, four subnets CLIENT, DMZ, SRV, SOC, interconnected via firewall fw1; more Windows hosts. SOC is unreachable to all agents to avoid interfering with logs. Time window is two hours, time step frequency is 1/30 seconds.
• Observations and actions: Wazuh generates alerts based on rules. The defender sends two commands via Active Response: shut down a specified machine, or block traffic bidirectionally between that host and other subnets on fw1; except for interactions with SOC. Each step, decide the action and target object.
• Language: The authors created CadsLang, a reduced version of CoreLang, with 9 asset classes, 49 attack steps, and 2 defense steps. The two attack paths are software vulnerabilities and credential brute force. notPresent blocks both paths on the host; restricted blocks access from other networks, still allowing internal subnet traffic.
• Mapping: Osquery data builds the MAL instance model; 11 attack steps are mapped to Wazuh rule IDs, rule groups, and rules to ignore. Without attacker and defender, the authors treat observed attack steps as false positives: without users, average false positive probability per time step, per attack step is 1.7%, with users 3.3%.

Cost and Decision Process

• Cost: Hosts are scored by confidentiality, integrity, and availability on a scale of 1 to 5. Data.read and Data.write count confidentiality plus integrity; notPresent counts availability, restricted counts half of availability. Single-step cost is the sum of attack and defense items.
• Return: The main metric is episode return, the discounted sum of per-step costs. On ADS-24, the authors define the moment Lore successfully executes a command with an elevated shell as Data.write and Data.read on the corresponding host; defense steps are considered active from when the command is sent to the Wazuh REST API.
• POMDP: The authors follow an episodic POMDP. Since transition probabilities are not assumed known, they use model-free deep RL. State consists of discrete entities and sparse logical relations in MAL; Vejde converts first-order logic facts into a bipartite factor graph, encodes with a message passing network, and the policy only looks at types, not specific identifiers.

Simulation Training and Range Interface

• Training: RL strategies are trained with MAL Simulator and Vejde, totaling 2 million transitions, each episode mixing different attacker strategies, entry points, and network topologies. Vejde w/ Noise is trained with 1% false positives and false negatives per attack step. Appendix A has training and simulation test details; the main text does not include the specific hyperparameters from that appendix.
• Heuristics: If an attack step associated with an asset is observed, choose the defense step associated with that asset. All four strategies share the same MAL input: Vejde, Vejde w/ Noise, Heuristic, NoOp.
• Online loop: Episodes start from the same snapshot, starting half an hour after recovery. The observation library is initialized with the instance model; query Wazuh every 30 seconds, write attack steps on match, merge duplicate instances. No action is requested when the database has no changes. Vejde takes the highest-probability action. After execution, write the defense step into the observation library and remove that asset and its associations; if the alert appears again later, temporarily put it back for one step.
• Scenario variables: whether to run simulated users; Lore uses Guided or Exploratory. Guided uses a blacklist to ignore machines not on the fastest path from flightlogs to payroll in the DMZ; Exploratory can prioritize attacking machines not closer to the DMZ, and has more actions. Topology changes randomly remove hosts from rootca, timereporter, print, which are not on Guided's target list.

Experimental Setup

• Tested strategies: Vejde; Vejde w/ Noise (1% false positives and false negatives per attack step during training); Heuristic; NoOp. RL strategies are trained in the MAL Simulator and then connected to ADS-24.
• Environment: ADS-24 in Crate, 37 virtual machines; red team is Lore, entry point flightlogs; optional simulated users. The running machine is an 11th-gen Intel i7, 4.800 GHz, integrated graphics.
• Scale: 134 episodes, collected over about a month, approximately 30 episodes per agent type. Each episode lasts two hours, starting half an hour after recovery, with alert queries every 30 seconds.
• Variables: presence or absence of simulated users; Lore's Guided and Exploratory modes; random removal of hosts from rootca, timereporter, print. Each episode draws one defensive agent, then returns it to the set after drawing.
• Metrics: Episode return R is the sum of per-step costs. The authors split R into attack cost Ra and defense cost Rd, and compute Rα = (1−α)Ra + αRd, with α from 0 to 1. α=0 considers only availability, α=1 considers only the attack side, availability not counted.
• Ground truth: Lore records successful and failed actions on the control plane for post-hoc evaluation. The paper does not report reviewer model or human agreement in the attached main text.

Main Results

Verifiable numbers given in the main text are as follows (the violin plot in Figure 6 is truncated in the attached text, and return values for each condition are not listed).

Strategy	Wait probability	Authors' summary of costs
Vejde	85%	Overall lowest attack cost, highest average defense cost
Vejde w/ Noise	94%	Highest average return, defense cost lower than Vejde and Heuristic
Heuristic	88%	Authors claim it can block Lore, but because it acts on all alerts, availability cost is high
NoOp	Not reported	Always best across configurations at α=0

• Authors' interpretation: The higher return of Vejde w/ Noise mainly comes from fewer asset shutdowns and lower availability cost; more permissive strategies let Lore into the client network in some episodes, leading to higher attack cost.
• Strategy differences: The authors consider Vejde close to Heuristic, but it can also act on events on associated assets, while Heuristic only acts on assets where attack steps directly appear. The lower the wait probability, the less conservative the authors describe it as.
• Weights: At α=0, NoOp is best because all defensive actions affect availability; at α=1, the authors claim Vejde is best in most configurations. The intersection of a strategy's return line with NoOp's line is the switching point the authors define as "better than doing nothing." The text after Figure 6 is truncated in the attached text, and specific intersection values for each configuration are not included.

False Positives in Alert Mapping

• What was tested: Collected multiple two-hour data runs without attacker and defender, with and without simulated users.
• Results: Without users, average false positive probability per time step, per attack step is 1.7%; with users, 3.3%.
• Authors' use: These observations are treated as false alarms and used to explain why a strategy variant was trained with 1% noise.

Lore Paths and Action Distribution

• Paths: The authors state that under Guided, Lore only moves toward machines in the DMZ; Exploratory is more dynamic, but most of the time it compromises the CLIENT segment. The transition probability graph is in Figure 10 of Appendix E; the attached text does not give the probability numbers from the figure.
• Actions: The authors report overall wait probabilities for Vejde, Heuristic, and Vejde w/ Noise as 85%, 88%, and 94%, respectively.
• Authors' interpretation: Lower wait probability corresponds to more aggressive handling; noise training corresponds to more conservative behavior and lower defense cost.

Other Ablations and Analyses

• Training budget: 2 million transitions, mixing attacker strategies, entry points, and topologies; in-simulation test results are in Appendix A, and the attached main text does not give simulation return numbers.
• Noise training: 1% false positives and 1% false negatives per attack step.
• False positive baseline: 1.7% without users, 3.3% with users (per time step, per attack step).
• Weight sweep: α∈[0,1], endpoint conclusions are NoOp and Vejde, intermediate values are not listed in the attached main text.
• Entry point fixed to flightlogs, and that host has no Wazuh agent; the authors state Lore cannot be fully expelled.
• Observation library occasionally delayed, responses lost in traffic; if a removed asset reappears in alerts, it is only temporarily put back for one step.

Limitations and Future Directions Noted by the Authors

• Still far from practical: At the end of the introduction, the authors conclude from the results that RL-based automated defense strategies have potential, but the method still requires substantial work to become practical (Introduction).
• Sim-to-real gap: The authors treat the sim-to-real gap in robotics as a known obstacle and state that models trained only in simulation or on datasets exhibit generalization and robustness issues on real systems; this is why they test in a range, rather than treating the gap as solved (Introduction).
• Simulation training can overfit: The authors cite Wolk et al. and Hicks et al., stating that network defense agents trained in simulation tend to overfit specific network topologies or attacker strategies, making them less practical under more operational conditions (Introduction).
• Cannot train directly on the target system: The authors state that model-free RL requires many samples, and direct training on the target problem may be infeasible, so they adopt simulation training followed by transfer (Related Work, Agent Training In Simulation).
• Cost of real-network testing: The authors state that testing against real threats on real networks lacks control, is costly, and makes it hard to know ground truth about attack progress; the cost of experimental autonomous agents failing or mishandling false positives makes this route uneconomical (Defender Agent Evaluation).
• Entry point cannot be cleared: flightlogs has no Wazuh agent, is invisible and inoperable to the defender agent, and the authors state that Lore can therefore never be fully expelled from the network (Network Activity). The attached main text has no separate Limitations section; future work is not written as a concrete experimental plan.

Experimental Coverage

• System: The evaluation network is ADS-24 in the Crate event plane, with 37 virtual machines and four subnets; the SOC is unreachable to the agent.
• Policies and episodes: Four policies, 134 two-hour episodes, about 30 per type; RL training uses 2 million transitions.
• Attacks and users: Lore has two configurations, Guided and Exploratory, with the entry point fixed at flightlogs; there are also variations with and without simulated users, and topology changes with random deletion from three specified hosts.
• Actions and observations: Defense commands are shutdown and bidirectional network disconnection; 11 attack steps are mapped to Wazuh; false positive rate is estimated in collections with no attacker and no defender.
• Not reported in the paper: The attached main text does not provide tables of returns, Ra, Rd values for each condition, does not report query counts, statistical tests for episode repetitions, nor specific returns from in-simulation tests; the tables and figures in Appendices A, D, and E are beyond the truncation, and their numbers are not included in this text.

The authors evaluate a simulation-trained autonomous network intrusion response agent in Crate, a training range for humans.

• Problem: The defender must prevent Lore from expanding access to ADS-24 within two hours, while minimizing the availability cost of shutdowns and network disconnections. The authors examine whether RL policies incur lower total cost than manual heuristics, and whether simulation-to-range transfer holds.
• Method: A reduced version of the MAL language, CadsLang, describes 9 asset types and two intrusion paths; Osquery builds instance models, and Wazuh alerts map to 11 attack steps. Vejde uses a graph neural network to output actions on a factor graph. Training is done in the MAL Simulator with 2 million transitions, mixing attacker strategies, entry points, and topology changes. Another variant is trained with 1% false positives and false negatives.
• Evaluation: 134 episodes, four policies. Cost attributes confidentiality and integrity to hosts where Lore obtains an upgraded shell, availability to defensive actions, and isolation counts as half the cost of shutdown.
• Results: The authors state that Vejde w/ Noise had the highest average return, because it shut down fewer assets and had lower defense cost than Vejde and Heuristic, but in some episodes it let Lore into the client network. Vejde had the lowest attack cost and the highest defense cost. The wait probability was 85% for Vejde, 88% for Heuristic, and 94% for Vejde w/ Noise. When only availability is counted, NoOp is best; when availability is not counted, the authors state that Vejde is best in most configurations. With no users and with users, the average false positive probability of the mapping is 1.7% and 3.3% per time step, per attack step.
• Conclusion: The authors state that RL automated defense policies have potential; Heuristic can stop Lore but at high availability cost, and performance depends heavily on the attacker strategy and simulated users; substantial work remains to reach practicality.