A systematic review of 100 papers: methods, applications and evaluation of LLM security agents

A review published on arXiv in August 2026 sorted through 100 papers published between January 2023 and March 2026 in 73 conferences and journals, using the three dimensions of methods, applications and evaluation to build a comparable taxonomy for LLM security agents. The study went through three stages—keyword search, forward and backward snowballing, and manual screening—converging from 108 candidates to 100 papers. Four findings include: single agents remain mainstream, while multi-agent systems diverge into three topologies: collaborative, hierarchical and peer-to-peer; applications cover 11 security task domains including vulnerability detection and penetration testing; evaluation mostly stays at outcome metrics such as task success rate, while process and trajectory metrics such as hallucination rate and tool-call correctness are covered by only some papers. The review argues that in this field action capability is in place first, permission governance lags behind, and evaluation practice comes last, and recommends that enterprises define permission boundaries and require process trajectory auditing before deployment.