CircuitSteer: geometrically aligned multi-layer steering based on sparse autoencoder circuits
CircuitSteer: Geometrically Aligned Multi-Layer Steering via Sparse Autoencoder Circuits
Translation in progress — untranslated text is omitted.
CircuitSteer uses sparse autoencoders (SAE) to identify and steer cross-layer semantic circuits, combining feature co-activation with the geometric alignment of decoding directions to build feature-flow circuits, then synthesizes dense steering vectors for multi-point intervention. In comparative sample evaluations on toxicity, emotional intensity, sycophancy and refusal tasks across two model families, it is the only method that consistently maintains text fluency; competing methods either sacrifice quality or have insufficient coverage, failing entirely on complex behaviors such as sycophancy and refusal.
- Inference-time behavior steering is a tool for AI alignment, but single-layer methods such as CAA apply the same intervention to semantically diverse inputs, and high-level concepts are distributed across layers, so single-layer interventions struggle to maintain consistent behavior change; naive multi-layer interventions in turn produce destructive interference due to unaligned feature directions, harming fluency.
- CircuitSteer uses a pretrained SAE to decompose the residual stream into sparse features, builds a cross-layer feature flow circuit using co-activation plus decoder direction cosine similarity, then filters out the target behavior subcircuit using a contrastive specificity score, averages the decoder directions of the features involved at each layer into a dense steering vector, and applies interventions simultaneously at multiple layers with a single coefficient λ.
- Compared against 8 baselines across the four datasets RTP, Jigsaw, Emotion, and Sycophancy on Gemma-2-2B and Llama-3.1-8B-Instruct, the authors claim CircuitSteer is the only method that maintains fluency effectively on all 8 configurations; ablations show that removing geometric alignment or swapping in random features causes failure on Sycophancy; the appendix reports that at λ=−3 it reduces Llama's refusal rate on AdvBench from 89% to 0%.
- The authors note that co-activation and geometric alignment are only approximate indicators of cross-layer causal influence, not causal guarantees; because publicly available multi-layer SAE suites stop at 27B, they did not evaluate at the 70B+ scale; when the compared prompt structures do not match, circuit discovery for Sycophancy fails (∆=0).