Scanning is not the same as AI red teaming: 94% of alerts are noise

Giskard used a bank payment Agent Aria running on GPT-5.4 mini to test four LLM security scanners: Garak, Promptfoo, DeepTeam, and PyRIT. Of 4,101 alerts, 94% required no fix; of 39 confirmed vulnerabilities, only 7 were detected, and of 9 severe vulnerabilities, only 4 were detected. The main cause of false positives was that the scanners' graders did not understand the Agent's purpose and judged normal behaviors such as asking about the payee as failures; in addition, 3 vulnerabilities were triggered but judged PASS by the scanners' own graders. Severe flaws were hidden in tool calls rather than text replies; the vulnerability of paying to a forged merchant ID was not found in any of 15,563 tests. Among 28 risk scenarios, the four scanners together implemented only 10 and never attempted 6, even though all claimed to cover the OWASP Top 10. The article suggests keeping the scanners and adding a threat model, using frameworks such as EBIOS Risk Manager and STRIDE to list risk scenarios, grading accordingly, counting coverage, and filling red-team gaps.