Benchmarking Jev against open-source decision model prompt injection detectors
Detector benchmark: Jev and the open decision models
This benchmark compares Jev 1.13.0, Kev-4B, Kev-0.8B, Laya, GLiNER2.5-Decide, ProtectAI DeBERTa, Prompt Guard 2, and keyword rules on the same Apple M4 machine across two task types: document chunk injection and user message jailbreak. In the document chunk test, Jev achieves an AUC of 1.00, 100% recall at a frozen threshold with 8% false positives, Kev-4B achieves 0.90 and 70%, and Prompt Guard 2 achieves 0.83 and 19%; on covert attacks that do not directly mention AI, Jev achieves 97% recall, Kev-4B 62%, and the three dedicated injection classifiers perform near random levels. On 220 real human attacks from LLMail-Inject, Jev catches 95% with 0% false positives, Kev-4B catches 86% with 1% false positives; when used as an email assistant, gpt-oss-120b sends emails to the attacker's address in 20/220 cases, and with Jev in front, 0 pass through.
这份评测把提示注入检测器放到网关真实任务上横向对比,并给出端到端攻击成功率,可用来判断各检测器的实际拦截差距。