Alignment Curse: Modality Alignment Greatly Enhances Audio Attacks via Text Transfer
The Alignment Curse: Modality Alignment Supercharges Audio Attacks via Text Transfer
Quick read
Based on 论文全文 （AI 生成）; refer to the original
The paper proposes the "Alignment Curse" principle, arguing that the stronger the text-audio modality alignment in omni models, the more effective text jailbreaks transfer to audio. The authors conduct black-box evaluations of 11 attacks on Qwen2.5-Omni-3B/7B, Qwen3-Omni-30B, InteractiveOmni-8B, and gpt-4o-audio-preview using JailbreakBench and AdvBench subsets, finding that TTS-converted text-to-audio transfer attacks reach or exceed native audio attacks on most models; under audio-only access, PAP(A) achieves an average SR of 0.71 and AutoDAN-Turbo(A) 0.58. Representation-layer KL is negatively correlated with transfer effectiveness, and the theoretical bound is nontrivial when KL is below 2; text-trained Linear/MLP Probes also transfer to audio, but with a performance gap.
- 端到端训练的 omni-model 通过加强文本-音频模态对齐提升了音频能力，但这种对齐是否会把文本模态的安全漏洞迁移到音频尚不清楚。文本越狱攻击远比音频攻击成熟，若系统性迁移，现有音频安全评估可能低估来自文本的风险。
- 作者提出并形式化刻画 Alignment Curse：用表示层 KL 散度衡量模态对齐，证明对齐越紧（KL 越小），文本攻击越容易迁移到音频。随后在 Qwen2.5-Omni、Qwen3-Omni、InteractiveOmni、gpt-4o-audio 等模型上做黑盒评估，比较文本攻击、经 TTS 转成音频的文本迁移攻击和原生音频攻击。
- 在 JailbreakBench 与 AdvBench 子集上，文本攻击平均成功率最高；文本迁移音频攻击（如 PAP(A)、AutoDAN-Turbo(A)）在多数模型上追平甚至超过专门的音频攻击，在仅音频访问下也占优。跨模型迁移中 PAP(A) 平均 SR 达 0.71。表示层 KL 与迁移效果呈负相关，KL 低于 2 时迁移更强；文本训练的探针也能部分迁移到音频。
- 未涉及图像、视频等更广模态，也未验证后门、偏见、幻觉等其他安全风险的跨模态迁移，实证仅限越狱攻击。防御迁移分析未考虑信号级扰动等音频特有攻击向量，作者建议未来扩展跨模态迁移与防御研究。
The paper links the degree of modality alignment with the transfer of text jailbreaks to audio and provides comparative data on 5 omni models, which can serve as a reference for audio safety evaluation.