Words beat code: a study of cognitive heuristics in LLM code vulnerability detection

Researchers found that eight large language models are all affected by cognitive heuristics in code vulnerability detection; merely changing the context surrounding the code can sway the judgment result. They propose a controlled framework that triggers the halo effect through author attribution, the framing effect through task goals and consequences, and the anchoring effect through prior analysis results, tested on three programming languages. Across models, the average susceptibility rate is highest for the framing effect at 33.2%, followed by the anchoring effect at 23.5% and the halo effect at 18.4%. Vulnerabilities that require semantic reasoning to detect are more susceptible than those identifiable through pattern matching; models often change their judgment from safe to vulnerable without accurately pointing out the real vulnerability. The authors also demonstrate a proof of concept for a black-box cognitive attack that can suppress up to 97% of previously detected vulnerabilities.