团队发布仅 46k 参数的提示注入防御模型
研究者 Alexander Panfilov 发布了一个仅 46k 参数的提示注入防御模型。该工作用基于 RL 和 Claudini 发现的攻击做了攻击者后手（attacker-moves-second）评估，作者称 Claudini 很适合用作自适应攻击评测。作者称所得模型比 SecAlign 更少出现角色混淆。作者还指出，模型中存在帮助识别指令的注意力头和促使模型执行指令的注意力头，增强前者可提升提示注入检测。合作者称其攻击显示 HoS 并非不可攻破，但比他们尝试过的其他方法更难攻破，并提供了 Hugging Face 在线演示、论文 arXiv 2610.37657 与代码仓库。
46k 参数的提示注入防御模型，用 RL 与 Claudini 自适应攻击做攻击者后手评估，并给出注意力头层面的机制线索。
We release a very cool (only 46k params) prompt injection defence!!
1) We did a proper attacker-moves-second exercise with RL-based and Claudini-discovered attacks: Claudini really works quite nicely as an adaptive attacks eval!
2) Resulting model is much less role-confused than SecAlign.
3) There are attention heads that help recognising instructions, and heads that make model execute instructions. Bumping former heads increases prompt injection detection!
Our attacks show that HoS isn't unbreakable, but it's a lot harder to break with everything we've tried. We'd love to see new attacks against HoS from the community!
Try HoS live on Hugging Face: https://huggingface.co/spaces/igorshilov/heads-of-security
Paper: https://arxiv.org/abs/2610.37657
Code: https://github.com/imperial-aisp/heads_of_security
Huge thank you to my amazing collaborators @_igorshilov, @kotekjedi_ml, @EKortukov, @matthieu_meeus, @romovpa, @WojciechSamek, @jonasgeiping, @yvesalexandre, I've really enjoyed this project!在 X 查看被引用的帖子