Promptfoo vs. Giskard: Red-Teaming an AI Payment Agent
Promptfoo vs Giskard: red teaming an AI payment agent
Giskard used the AI payment agent Aria, running GPT-5.4 mini, as a target, and measured the recall and false positives of Promptfoo's default red-team scan against 42 vulnerabilities confirmed by its own audit (9 critical; counted as 39 after excluding 3 confabulated confirmations). Promptfoo's 1,365 single-turn tests reported 147 failures, of which 91 corresponded to confirmed vulnerabilities and 22 (15%) were false positives, but it detected only 5 of the 39 vulnerabilities and 3 of the 9 critical vulnerabilities, and 1 additional vulnerability was triggered but judged PASS by its grading model. Of the 28 risk scenarios in Aria's threat model, 9 were never attempted, including concurrent payments and impersonating customer-service fraud; the authors note that the key harm lies in tool calls rather than response text, for example paying a forged merchant ID.