Syntax- and Compilation-Preserving Evasion Attacks on LLM Vulnerability Detectors
Syntax- and Compilation-Preserving Evasion of LLM Vulnerability Detectors
Quick read
Based on 论文全文 （AI 生成）; refer to the original
The paper proposes attack methods that evade LLM vulnerability detectors while preserving syntax and compilation, evaluating six detection models on a unified C/C++ benchmark (N=5000), with five of them achieving a complete resistance rate (CR) no higher than 13%, meaning over 87% of correctly detected vulnerabilities can be evaded by at least one syntax-preserving edit. The authors construct five attack variants using four carrier families (identifier replacement, comments, preprocessor macros and #ifdef, dead branches), and universal adversarial strings optimized on a 14B proxy model transfer with zero modification to black-box APIs like GPT-4o, with targeted optimization achieving an attack success rate of up to 99.88%. On the defense side, input sanitization only removes comment and preprocessor carriers, randomized smoothing offers limited gains (1.6–8.9 percentage points), and adversarial fine-tuning raises CR from 0.13% to 3.88% but reduces clean TPR by 6.3 percentage points.
- LLM 漏洞检测器正被大规模部署到 CI/CD 安全门禁中，但它们对保持语法与编译的规避编辑是否稳健尚不清楚。若攻击者能用重命名标识符、加注释、插死代码等无害变换让已标记的漏洞代码通过扫描，自动安全门禁就会失效。
- 作者在统一的 C/C++ 基准（N=5000）上评估四类载体、五种攻击变体，用 GCG 在载体约束下优化通用对抗字符串，并引入 Complete Resistance（CR）衡量能同时抵抗全部攻击变体的比例。评估覆盖白盒、代理迁移黑盒 API 和受限灰盒三种访问假设。
- 六个模型中五个的 CR ≤ 13%，即超过 87% 被正确检出的漏洞可被至少一种保语法编辑规避；CodeAstra 在定向攻击下 CR 仅 0.12%。在 14B 代理上优化的通用字符串可原样迁移到 GPT-4o 等黑盒 API，定向优化最高达 99.88% ASR。三种防御均不充分：清洗只消除注释与预处理载体，随机平滑仅降 1.6–8.9 个百分点，对抗微调把 CR 从 0.13% 提到 3.88% 但清洁 TPR 下降 6.3 个百分点。
- 仅限 C/C++ 与固定提示协议；验证只保证语法与编译层面保持，未证明运行时等价。优化出的字符串常不自然，可能被风格检查或命名规范拦截，而攻击成功率假设没有这类预筛。多随机种子分析只覆盖定向标识符替换，更广的载体与访问机制留待后续。
The paper uses syntax- and compilation-preserving code transformations to quantify the robustness gap of LLM vulnerability detectors despite high scores on clean benchmarks, providing a reference for teams deploying detection gates.