PentestGPT: an LLM-driven automated penetration testing tool

The researchers proposed and open-sourced PentestGPT, an LLM-driven automated penetration testing tool, which on their self-built benchmark achieved a 228.6% improvement in task completion rate over the LLMs used for comparison. The benchmark was built from test machines equipped with various platforms, to evaluate LLM performance on real penetration testing tasks. The study found that LLMs can complete subtasks such as using testing tools, interpreting outputs and proposing next actions, but struggle to maintain an overall understanding of the entire testing scenario. PentestGPT accordingly designs three self-interaction modules, each corresponding to a different penetration testing subtask, to mitigate the context loss problem. The tool has been open-sourced on GitHub and has received more than 4700 stars.