HoS: Learned Attention Gating Defense Against Prompt Injection

1. 00:30
   X@Iam_Shivanshu· Discussion

   97% → 10%. That's the best RL-trained prompt-injection attacker's success rate on GPT-OSS-20B before and after a defen…(opens the original post in a new tab)

1. 02:10
   X @kotekjedi_mlSelected

   Team releases prompt injection defense model with only 46k parameters

   Researcher Alexander Panfilov released a prompt injection defense model with only 46k parameters. The work used attacker-moves-second evaluations with attacks based on RL and attacks discovered by Claudini, and the authors say Claudini is well suited for use in adaptive attack evaluation. The authors say the resulting model exhibits less role confusion than SecAlign. The authors also note that the model contains attention heads that help identify instructions and attention heads that push the model to execute instructions, and that strengthening the former can improve prompt injection detection. Collaborators say their attacks show HoS is not unbreakable, but is harder to break than other methods they tried, and they provide a Hugging Face online demo, the paper arXiv 2610.37657, and a code repository.

2. 00:22
   X@xiaoxue__yang· Discussion

   Our attacks show that HoS isn't unbreakable, but it's a lot harder to break with everything we've tried. We'd love to s…(opens the original post in a new tab)
3. 00:22
   X@xiaoxue__yang· Discussion

   Caveat: HoS enforces an externally supplied trust boundary. It doesn't decide what's trusted, and it's not designed for…(opens the original post in a new tab)
4. 00:21
   X@xiaoxue__yang· Discussion

   📣 New paper: Heads of Security (HoS)! Can 18K parameters defend agents against prompt injection without sacrificing ut…(opens the original post in a new tab)
5. 00:17
   X@_igorshilov· Discussion

   Want to see more? We have more cool results on GPT-OSS-20B and Llama-3.3-70B Check out our resources: Full paper: htt…(opens the original post in a new tab)
6. 00:17
   X@_igorshilov· Discussion

   There are of course caveats. 1) We aim for instruction-data separation. It is useful in some scenarios (e.g. summarizi…(opens the original post in a new tab)
7. 00:17
   X@_igorshilov· Discussion

   I'm very excited to share our new paper on prompt injection defences: "Heads Of Security"! Inspired by interp work sho…(opens the original post in a new tab)

1. 05:58
   X@okey_amy· Discussion

   Prompt injection defenses usually mean more prompts. Heads of Security learns per-head attention gates that enforce ins…(opens the original post in a new tab)

1. 21:13
   X@pryzmat_tech· Discussion

   3/7 RL-Hammer hits 97% ASR on GPT-OSS-20B, 85% on SecAlign++, only 10% on HoS. White-box Claudini needs 3x more FLOPs…(opens the original post in a new tab)
2. 21:13
   X@pryzmat_tech· Discussion

   2/7 Heads of Security: Learned Attention Gates Improve Prompt Injection Robustness arXiv:2610.37657 Learned per-head…(opens the original post in a new tab)
3. 15:14
   X@bravish_· Discussion

   Imperial, the Tübingen AI Center / ELLIS and Max Planck trained 46,080 numbers to make a 70B model far harder to prompt…(opens the original post in a new tab)
4. 08:42
   X@robinliu2022· Discussion

   4/6 Heads of Security: learned attention gates to prevent prompt injection. Learn a gate for each attention head, reallocating attention between trusted/untrusted tokens to achieve instruction-data separation; on Llama-3.3-70B and GPT-OSS-20B compared with SecAlign…(opens the original post in a new tab)

1. 01:58
   arXiv: Jailbreaks, Prompt Injection, Poisoning & Defenses

   Heads of Security: Learned Attention Gating Improves Prompt Injection Robustness

   Researchers propose Heads of Security (HoS), a prompt injection defense that enforces instruction-data separation by learning each attention head's attention allocation between trusted and untrusted tokens. The authors compare it with the fine-tuning-based SecAlign++ and with ASIDE, which likewise implements instruction-data separation at the architectural level, on Llama-3.3-70B-Instruct and GPT-OSS-20B: the black-box attack RL-Hammer achieves a 97% attack success rate on GPT-OSS-20B and 85% on SecAlign, but only 10% on HoS; for the white-box attack Claudini, strengthened against HoS via autoresearch, the FLOPs required to reach the same attack success rate on HoS are 3 times those on GPT-OSS-20B or SecAlign.

1. Date not given
   AI Security and Privacy Lab, Imperial College LondonSelected

   Heads of Security: Learned attention gating improves prompt injection robustness

   Researchers from Imperial College London and other institutions proposed the Heads of Security (HoS) defense method, which trains learned gates on each attention head so that the model can read tool outputs without executing instructions injected into them. Each token is labeled as trusted, untrusted, or template channel, and each head scales attention scores by channel before softmax, thereby preventing instruction-following heads from reading untrusted content while other heads can still process that content. The method trains only gate parameters: 18,432 scalars on GPT-OSS-20B and 46,080 on Llama-3.3-70B, trained with the SecAlign++ DPO recipe, with base weights unchanged.

Current attention 38·Peak in the comparable period 38 (Nov 6 03:00)·Change over the comparable past 24 hours 0%