Incalmo: An Autonomous LLM-assisted System for Multi-host Network Red Teaming

The authors propose Incalmo, an LLM-assisted system for autonomously conducting red teaming against multi-host networks, and it successfully obtained key assets in 37 of the 40 simulated networks in their self-built benchmark MHBench. Incalmo has an LLM plan high-level declarative tasks, hands them to domain task agents for execution, and uses auxiliary services to manage context and acquired assets. By comparison, existing LLM-assisted systems succeeded in only 3 MHBench environments; the authors also find that PentestGPT and CyberSecEval3 paired with models such as Sonnet 4 and Gemini 2.5 Pro cannot autonomously complete multi-host red teaming exercises. MHBench contains 40 networks simulating real environments, ranging in scale from 22 to 50 hosts; successful attacks took 12 to 54 minutes, with LLM call costs below 15 US dollars.