Skip to content

CircuitSteer: geometrically aligned multi-layer steering based on sparse autoencoder circuits

CircuitSteer: Geometrically Aligned Multi-Layer Steering via Sparse Autoencoder Circuits

arXiv· arXiv:2608.05732· Mehrshad Saadatinia, Parsa Razmara, Ardalan Aryashad, Ali Abbasi, Seyedarmin AziziUniversity of Southern CaliforniaIndexed by this site · 19 days ago· Originally publishedAlignment

Translation in progress — untranslated text is omitted.

Quick read

Defenses

Based on the paper PDF (AI-generated); refer to the original

AI summaryCircuitSteer uses sparse autoencoders (SAE) to identify and steer cross-layer semantic circuits, combining feature co-activation with the geometric alignment of decoding directions to build feature-flow circuits, then synthesizes dense steering vectors for multi-point intervention. In comparative sample evaluations on toxicity, emotional intensity, sycophancy and refusal tasks across two model families, it is the only method that consistently maintains text fluency; competing methods either sacrifice quality or have insufficient coverage, failing entirely on complex behaviors such as sycophancy and refusal.

Questions
Inference-time behavior steering is a tool for AI alignment, but single-layer methods such as CAA apply the same intervention to semantically diverse inputs, and high-level concepts are distributed across layers, so single-layer interventions struggle to maintain consistent behavior change; naive multi-layer interventions in turn produce destructive interference due to unaligned feature directions, harming fluency.
Methods
CircuitSteer uses a pretrained SAE to decompose the residual stream into sparse features, builds a cross-layer feature flow circuit using co-activation plus decoder direction cosine similarity, then filters out the target behavior subcircuit using a contrastive specificity score, averages the decoder directions of the features involved at each layer into a dense steering vector, and applies interventions simultaneously at multiple layers with a single coefficient λ.
Experiments and results
Compared against 8 baselines across the four datasets RTP, Jigsaw, Emotion, and Sycophancy on Gemma-2-2B and Llama-3.1-8B-Instruct, the authors claim CircuitSteer is the only method that maintains fluency effectively on all 8 configurations; ablations show that removing geometric alignment or swapping in random features causes failure on Sycophancy; the appendix reports that at λ=−3 it reduces Llama's refusal rate on AdvBench from 89% to 0%.
Limitations and future work
The authors note that co-activation and geometric alignment are only approximate indicators of cross-layer causal influence, not causal guarantees; because publicly available multi-layer SAE suites stop at 27B, they did not evaluate at the 70B+ scale; when the compared prompt structures do not match, circuit discovery for Sycophancy fails (∆=0).

Related papers

Show 3 more papers
Read originalarxiv.org