Team releases prompt injection defense model with only 46k parameters
Researcher Alexander Panfilov released a prompt injection defense model with only 46k parameters. The work used attacker-moves-second evaluations with attacks based on RL and attacks discovered by Claudini, and the authors say Claudini is well suited for use in adaptive attack evaluation. The authors say the resulting model exhibits less role confusion than SecAlign. The authors also note that the model contains attention heads that help identify instructions and attention heads that push the model to execute instructions, and that strengthening the former can improve prompt injection detection. Collaborators say their attacks show HoS is not unbreakable, but is harder to break than other methods they tried, and they provide a Hugging Face online demo, the paper arXiv 2610.37657, and a code repository.
A 46k-parameter prompt injection defense model, using RL and Claudini adaptive attacks for attacker follow-up evaluation, and providing mechanistic clues at the attention head level.
We release a very cool (only 46k params) prompt injection defence!!
1) We did a proper attacker-moves-second exercise with RL-based and Claudini-discovered attacks: Claudini really works quite nicely as an adaptive attacks eval!
2) Resulting model is much less role-confused than SecAlign.
3) There are attention heads that help recognising instructions, and heads that make model execute instructions. Bumping former heads increases prompt injection detection!
Our attacks show that HoS isn't unbreakable, but it's a lot harder to break with everything we've tried. We'd love to see new attacks against HoS from the community!
Try HoS live on Hugging Face: https://huggingface.co/spaces/igorshilov/heads-of-security
Paper: https://arxiv.org/abs/2610.37657
Code: https://github.com/imperial-aisp/heads_of_security
Huge thank you to my amazing collaborators @_igorshilov, @kotekjedi_ml, @EKortukov, @matthieu_meeus, @romovpa, @WojciechSamek, @jonasgeiping, @yvesalexandre, I've really enjoyed this project!View the quoted post on X