Depth-Aware Activation Steering Based on Attention Residuals
Depth-Aware Activation Steering via Attention Residuals
On two small open-source models, Qwen3-0.6B and Qwen3.5-0.8B, researchers combine the cross-layer pathways of Attention Residuals with Jacobian-lens concept readout to propose a low-overhead depth-aware activation steering pipeline. Distributing persona vectors by depth weight yields stronger deep-layer concept alignment (p<0.0001) and behavioral shift than single-layer injection; the effect is sign-specific, with the reversed vector flipping all six metrics, random directions reducing alignment, brief LoRA fine-tuning eliminating the imprint, and injection strictly reshaping attention only downstream of the injection point (upstream JS=0.00000). Training depth weights on the steering objective (200 steps, frozen backbone) increases final alignment on held-out prompts 10-fold, with 96-99% of the effect retained across persona vectors; code and data are publicly available.