Skip to content
Giskard·Indexed by this site · Originally published· Evaluation →· Its place in the scan record →

Giskard Tests the PyRIT Red-Teaming Framework: 0 of 39 Confirmed Agent Vulnerabilities Detected

PyRIT vs Giskard: 0 of 39 Agent Vulnerabilities Detected

AI summary

Security vendor Giskard tested the default configuration of Microsoft's open-source red-teaming framework PyRIT and found that it detected 0 of 39 audit-confirmed agent vulnerabilities (including 9 critical vulnerabilities), and 8 of the 9 failures it reported were false positives. The test target, Aria, is a payment agent running GPT-5.4 mini with 8 tools; Giskard launched 520 tests against 20 default HarmBench goals using 25 Foundry techniques. The false positives stem from the default scorer counting any non-refusal response as a successful attack; PyRIT triggered the agent 47 times to print an account snapshot containing the payment token AGT-9F2B-77401, yet judged it PASS each time. The default goals barely overlap with the 28 risk scenarios in the agent's threat model, and PyRIT never attempted 25 of them. The authors recommend keeping PyRIT but supplementing it with a threat model, setting an attack goal and corresponding scorer for each scenario; the text states that the reference benchmark is Giskard's own audit and that PyRIT was run with its default configuration.

Read originalgiskard.ai