AWS tests AI agent autonomously finding and exploiting vulnerabilities, 33 of 37 CVEs succeeded

AWS published the test results of its autonomous penetration testing AI agent AWS Security Agent. Of 37 CVEs disclosed between 2025 and 2026, it successfully exploited 33 and obtained test flags, a success rate of 89.2%. The test used the Claude Opus 5 model, with a 10-hour time limit per vulnerability, conducted in an isolated environment in a manner similar to a capture-the-flag competition, covering types such as command injection, XSS, SQL injection and SSRF, with some cases requiring multiple attack steps to be completed. To prevent the model from memorizing publicly known exploitation methods, the researchers built their own test set and disabled the CVE lookup function during testing. Initially 35 obtained flags; after item-by-item review, 2 of these were found to have exploited design flaws in the test environment rather than the specified vulnerabilities, and 33 were ultimately deemed successful. Another 2, although they successfully exploited the vulnerabilities and executed code, caused the application or service to stop operating due to subsequent operations and failed to obtain flags; AWS stated that this test did not enable the non-destructive protection restrictions used in formal customer penetration tests.