SelfOp: An Optimization Algorithm for Self-Improving Security Agents

The researchers propose SelfOp, an algorithm that automatically optimizes the task context (instructions, skills, and reference documents) of security agents without modifying the execution framework or model weights, targeting security tasks such as vulnerability discovery, vulnerability reproduction, and patch generation. The method models context optimization as a chain-rule-inspired textual gradient descent: starting from the result of a single instance, it backpropagates the error signal through the evaluator, the agent trajectory, and the context artifacts to obtain per-instance textual gradients, which are then accumulated across instances via clustering, ranking, and filtering, and committed only when consistent across instances; a convergence detector monitors the gradient signal itself, allowing stopping without held-out validation data. On the real-world vulnerability reproduction benchmark CyberGym, using fewer than 200 training samples, SelfOp improves GPT-5.4-mini (with Codex) by 17 points, enough to exceed the frontier GPT-5.4 baseline by 6 points, and improves GPT-5.4 itself by 18.5 points; the optimized skills can also transfer across models.