Beneficial or Harmful? Evaluating LLM-Assisted Vulnerability Repair Through Human Experiments

The researchers designed a controlled experiment with human participants comparing LLM-assisted vulnerability repair with manual debugging, and have completed a small-sample pilot. The study hypothesizes that although LLM assistance may speed up repair, it is also prone to producing superficial fixes that bypass functional checks but fail security validation due to hallucinations or insecure code. The experiment uses a balanced crossover design, testing patches with a WebApp that integrates hidden Ghost Tests, and measures repair speed, repair effectiveness, and participant perceptions.