Confining the Model, Missing the System: Measurement and Accountability in Offensive AI Governance

Existing tools for measuring AI attack capabilities overstate harm and misattribute capabilities belonging to surrounding systems to the model; the paper argues that policy and procurement should adopt system-level, actual-harm-based capability evaluations. The research is set against the background of two frontier models being suspended in June 2026 due to US export controls, reportedly triggered by a jailbreak that had a model read a codebase and fix defects. The authors experiment on an open-source framework in which lightweight LLM agents collaborate through shared memory and evolutionary optimization: in swarm-generated attacks exceeding 225 per target, LLM-as-judge scoring judged Claude Sonnet 4 to be compromised in 40% of attacks, while manual verification found no content that could actually cause harm, whereas GPT-4o had an effective harm rate of 45.8%; on targets with implanted vulnerabilities, a complete pipeline built around a 1.2B-parameter model recovered all 9 weaknesses, while the same model with manual components removed recovered 0 of 9 by crash verification and 2 by referenced source lines.