Representation Engineering: A Top-Down Approach to AI Transparency

Core problem: the internal mechanisms of deep neural networks, especially LLMs, lack transparency, and existing bottom-up methods that use neurons and circuits as units of analysis struggle to explain and control high-level cognitive phenomena such as honesty, harmlessness, and power-seeking.

• Problem setting and importance: The authors note that LLMs are being deployed in domains such as healthcare, education, and social interaction, yet little is known about how they work internally and they can only be treated as black boxes; greater transparency helps understand decisions, assign accountability, and uncover hazards such as spurious correlations or hidden capabilities (Section 1).
• Shortcomings of existing methods: The authors argue that mechanistic interpretability corresponds to the Sherringtonian view in cognitive neuroscience, using connections between neurons as the unit of analysis; it has succeeded in explaining simple mechanisms but is difficult to scale to more complex phenomena; moreover, identifying circuits requires substantial manual effort, and evidence such as ResNet's robustness to whole-layer removal is incompatible with purely circuit-based explanations (Section 2.2).
• Core observation and hypothesis: The authors draw on the Hopfieldian view—that cognition is a product of representation space, realized by activity patterns of neuronal populations; the paper hypothesizes that many safety-relevant concepts and functions emerge naturally in LLM representations, and can therefore be read and manipulated directly at the representation level (Sections 1, 2.1).
• What the paper proposes: The authors propose representation engineering (RepE), a top-down transparency research approach, provide the representation reading baseline LAT and several representation control baselines (Reading Vector, Contrast Vector, LoRRA), and conduct preliminary analyses on safety-relevant issues including honesty, utility, morality and power, emotion, harmlessness, fairness, knowledge editing, and memorization (Sections 3–6).

Emergent structure in representations

• Mikolov et al. (2013), Bolukbasi et al. (2016), Schramowski et al. (2019)—semantic compositionality, gender bias, and moral dimensions appear in word embeddings and text embeddings; Radford et al. (2017) found that training to predict the next token gives rise to sentiment-tracking neurons.
• McGrath et al. (2022) found that chess-playing RNNs acquire human chess concepts; in vision, self-supervised and generative training give rise to structures such as semantic segmentation, local coordinates, and depth tracking (Caron et al., 2021; Oquab et al., 2023; Karras et al., 2021; Chen et al., 2023).

Interpretability methods

• Saliency maps (Simonyan et al., 2013, etc.)—highlight regions in the input that the network attends to; the paper notes that their reliability has been questioned (Adebayo et al., 2018, etc.) and that they provide limited insight into internal representations.
• Feature visualization (Erhan et al., 2009; Nguyen et al., 2016, etc.)—construct high-activation inputs to explain neurons; the paper notes that it does not account for the distributed nature of representations.
• Mechanistic interpretability (Olah et al., 2020; Olsson et al., 2022; Wang et al., 2023a; Lieberum et al., 2023)—reverse-engineers networks into circuits; the paper notes that it requires substantial manual effort and is incompatible with evidence of iterative refinement and self-repair in ResNet/LLMs (Veit et al., 2016; McGrath et al., 2023).

Concept localization and editing

• Concept localization and probes: Network Dissection (Bau et al., 2017), TCAV (Kim et al., 2018), linear classification probes (Guillaume Alain, 2017; Belinkov, 2022); concept editing in the latent space of image generation models (Bau et al., 2019; Shen et al., 2020, etc.).
• Fact localization and editing in LLMs (Meng et al., 2023a;b; Hernandez et al., 2023) and concept erasure (Belrose et al., 2023b; Ravfogel et al., 2023, etc.).
• Truthfulness and activation steering: Burns et al. (2022)'s CCS uses logical consistency to find truthfulness representations; Azaria & Mitchell (2023) train classifiers on hidden layers to identify whether statements are true or false; Li et al. (2023c) use activation editing to improve truthfulness; Turner et al. (2023e)'s ActAdd uses the activation difference vector from a single stimulus to steer behavior; Rimsky (2023a;b;c) uses activation engineering for red-teaming and reducing sycophancy.

Baselines and benchmarks: The paper uses an ActAdd variant as the control baseline on TruthfulQA MC1 (Table 2), CCS as the baseline on encoder tasks (Table 10), and in utility experiments compares linear models such as PCA, K-Means, Mean Difference, Logistic Regression, and Prompt Difference (Figure 12).

The authors position this paper as: building on these works to propose an improved RepE method, whose difference from activation engineering is that it obtains insights from both reading and control experiments, and introduces representation tuning (LoRRA) (Sections 1, 2.3).

The overall approach is to use representations (rather than neurons or circuits) as the unit of analysis, divided into two parts, Representation Reading and Representation Control, each with baseline methods (Section 3).

Representation reading: LAT baseline

• Three-step procedure: Linear Artificial Tomography (LAT) mimics neuroimaging and consists of (1) designing stimuli and tasks, (2) collecting neural activity, and (3) constructing a linear model (Section 3.1.1, Figure 4).
• Concept task template: To elicit declarative knowledge, present the model with stimuli of varying degrees of the concept and ask about that concept; the paper defaults to unsupervised throughout, using no labels, and holds that unlabeled or model-self-generated stimuli can avoid annotation bias (Section 3.1.1).
• Function task template: To elicit procedural knowledge, design experimental tasks that require executing the target function (e.g., honesty) and reference tasks that do not, defaulting to using general instruction-tuning data (Taori et al., 2023) as stimuli; function analysis focuses on chat models.
• Neural activity collection: For concepts, take the representation at the −1 token position (default: the last token) in the task template, A_c = {Rep(M, T_c(s_i))[−1] ∣ s_i ∈ S}; for functions, take the representation at each token position of the model's response, because generating each new token requires invoking that function (Equations 1–2; Figure 5 shows that accuracy for concept tokens and the last token is highest in the middle-to-late layers).
• Linear model construction: Default unsupervised PCA, performing principal component analysis on paired neural activity difference vectors and taking the first principal component as the reading vector v, with prediction using the dot product of the representation and v; stimulus sets typically range from 5 to 128 items (Section 3.1.1, Appendix C.1).

Methods for evaluating reading results

• Four types of experiments: Correlation (examine prediction accuracy inside and outside the distribution), Manipulation (verify causal effects), Termination (measure performance decline after removal, analogous to lesion studies), Recovery (measure recovery after re-introducing what was removed, analogous to genetic rescue experiments) (Section 3.1.2).
• Level of evidence: The authors emphasize that LAT only provides correlational evidence, and causal or stronger conclusions require the latter three types of experiments; the paper subsequently mainly conducts correlation and manipulation experiments.

Representation control: three types of controllers

• Reading Vector: Directly uses the vector obtained by LAT, but it is independent of the stimulus and perturbs all inputs in the same direction; the authors believe this may make its control effect weaker.
• Contrast Vector: Forward the same input separately with a pair of contrastive prompts and take the difference between the two representations; it varies with the input, and the authors call it a stronger baseline, but it requires extra computation at inference, and modifying multiple layers simultaneously requires recomputation layer by layer starting from the earliest layer to avoid cascading effects (Algorithm 1, line 10).
• LoRRA: Low-Rank Representation Adaptation, attaching low-rank adapters to the attention weights (query and value) and fine-tuning with a representation objective constructed from contrast vectors (Algorithm 1); the controller can be merged into the model with no extra inference overhead.

Control operators and success criteria

• Three operators: Linear combination R′ = R ± v (stimulate or suppress); piece-wise operation R′ = R + sign(Rᵀv)v (conditional amplification); projection R′ = R − (Rᵀv/‖v‖²)v (remove the component along the v direction) (Section 3.2.1).
• Criteria: Reading effects use classification accuracy; control effects use behavior-level counterfactual changes (e.g., whether it lies, whether it refuses, whether completions reproduce the original text), and some experiments use a sentiment classifier as a proxy score (Section 5.1.1).

Experimental Setup

• Models tested: LLaMA-2-Chat-7B/13B/70B (main experiments), LLaMA-2-7B Base (Appendix B.8), Vicuna-13B (harmlessness), Vicuna-33B-Uncensored (honesty and morality monitoring), Vicuna-33B (generating consequences), microsoft/deberta-xxlarge-v2-mnli (encoder tasks), openai/clip-vit-base-patch32 (emotion, Appendix B.5).
• Datasets/benchmarks: TruthfulQA (MC1), ARC-e/ARC-c, OpenbookQA, CommonSenseQA, RACE, ETHICS (Utilitarianism and Commonsense Morality), the power dataset of Pan et al. (2023) and MACHIAVELLI (30 test games), AdvBench 64 harmful instructions + ShareGPT 64 benign instructions, the 500 harmful instructions of Zou et al. (2023), StereoSet, Alpaca, Ferg-DB, over 1200 emotion scenarios generated by GPT-4, over 100 manually curated half-completed quotes.
• Baselines: zero-shot standard evaluation and heuristic scoring (Table 1), few-shot prompting (Table 9), CCS (Table 10), ActAdd variants (Table 2), random vectors and positively added vectors (Table 7).
• Metrics and judgments: classification accuracy; TruthfulQA uses answer log probabilities; emotion experiments use a proxy sentiment classifier; harmlessness uses helpful rate (frequency of complying with benign instructions) and harmless rate (frequency of refusing harmful instructions); memorization uses Exact Match and Embedding Similarity.
• Number of repetitions: TruthfulQA LAT results are the mean and standard deviation of 15 trials (Table 8); QA benchmark LAT is the mean of 10 trials (Appendix B.1).

Main Results

TruthfulQA MC1 accuracy (%, according to Table 1 and Table 2):

Swipe horizontally to read the table

Model	Standard	Heuristic	LAT (S1/S2/S3)	ActAdd	Reading	Contrast	LoRRA
7B-Chat	31.0	32.2	55.0/58.9/58.2	33.7	34.1	47.9	42.3
13B-Chat	35.9	50.3	49.6/53.1/54.2	38.8	42.4	54.0	47.5
70B-Chat	29.9	59.2	65.9/69.8/69.8	not reported	not reported	not reported	not reported

• The authors state that LAT exceeds both zero-shot methods, and that three stimulus sources (as few as 10 examples) yield similar performance, indicating that a consistent concept of truthfulness exists within the model; the authors believe larger models have a better internal truthfulness model, and that low zero-shot accuracy largely stems from the model's dishonest "knowing but doing otherwise" (Section 4.1).
• The authors state that Contrast Vector achieves state-of-the-art on TruthfulQA, but inference compute overhead exceeds 3 times; LoRRA overhead is negligible (Table 2 note); the authors state that after control, the 13B model approaches GPT-4's performance on this dataset.
• The introduction states that honesty control improves TruthfulQA zero-shot accuracy by 18.1 percentage points and surpasses all prior methods.

Honesty Monitoring and Lie Detection

• What was tested: using true statements from Azaria & Mitchell (2023) as stimuli to extract an honesty function vector, token-level monitoring was performed on honest/lying scenarios (Figure 8, 9, 23).
• Results: the reading vector achieves over 90% accuracy in distinguishing held-out samples instructed to be honest or dishonest; the detector sums the middle 20 layers and can flag intentional lying, hallucination, and misleading expressions; in the grade scenario, when the model outputs D-, the logits show probabilities for A/B/C/D of 11.3%, 11.6%, 37.3%, 39.8% respectively (Section 4.3.2).
• Authors' interpretation: the authors believe honesty/dishonesty tendencies are distributed in LLMs, and the detector also flags thought processes related to lying, which makes lie detection more complex, with comprehensive evaluation left for future research.

Utility, Morality, and Power

• Linear model comparison (Figure 12): six linear models are compared on ETHICS Utility; the authors report that Logistic Regression has the highest correlation but almost no steering effect, while PCA, K-Means, and Mean Difference perform in a balanced way across the three types of experiments: correlation, steering, and termination.
• LAT five-task accuracy (Table 11): Utility 81.0, Morality 85.0, Power 72.5, Probability 92.6, Risk 90.7.
• MACHIAVELLI (Table 3): LoRRA is used to control immorality and power-seeking for LLaMA-2-Chat-7B/13B; on 7B, −Control reduces Power from 106.2 to 100.0 and Immorality from 100.2 to 93.5, while Reward slightly decreases from 19.5 to 19.4; on 13B, −Control reduces Immorality from 96.6 to 92.4 and increases Reward from 17.7 to 18.8. The authors state that negative control reduces immorality and power-seeking scores, with little interference to game Reward.
• Concept compositionality (Figure 16): risk is synthesized from extracted probability and utility concepts according to a formula, and is positively correlated with the directly extracted risk concept, more pronounced in early layers; the authors state this suggests a coherent internal structure in concept representations.

Emotion, Harmlessness, Fairness, Editing, and Memorization

• Emotion (Figure 18, Table 4): t-SNE shows that six emotions gradually form separate clusters in the middle-to-later layers of LLaMA-2-Chat-13B; after adding a Happiness vector to the model, the compliance rate on 500 harmful instructions rises from 0.0% to 100.0%, while adding Sadness remains 0.0%; the authors believe emotion steering could be used to bypass alignment.
• Harmlessness (Table 5): on Vicuna-13B, the harmfulness vector achieves over 90% classification accuracy on held-out harmful/benign instructions, and remains over 90% under manual jailbreaks and GCG adversarial suffixes; piece-wise control under GCG attack raises harmless rate from 16% to 83%, with helpful rate 92% (average 87.2%), whereas linear combination over-refuses benign instructions (helpful rate drops to 86 out of 92.5%).
• Fairness (Table 6, Figure 25): after negative control using only a vector extracted from race-bias stimuli, when LLaMA-2-Chat-13B generates sarcoidosis cases, the mention rate of women drops from 97.0% to 55.0%, and of Black women from 60.0% to 13.0%; the authors state this suggests what was extracted is a more unified bias representation.
• Knowledge editing (Figure 21): using a LAT vector to change "the Eiffel Tower is in Paris" to "is in Rome", the model maintains the modification under new phrasings and does not affect the Louvre's location; the authors state this demonstrates generalization and specificity.
• Memorization (Table 7): on the quote completion task, after subtracting the memorization direction, EM drops from 89.3% to 47.6% (LATQuote), and SIM drops from 96.8% to 69.9%; random vectors or positively added vectors have almost no effect; accuracy on historical event years slightly drops from 97.2% to 96.2%, and the authors state this vector corresponds to verbatim memorization rather than world knowledge.

Other Ablations and Analyses

• QA benchmarks (Table 9): LAT averages 59.2–86.4 across five benchmarks, mostly exceeding few-shot, with exceptions being RACE for 7B (45.9 vs 46.2) and CSQA for 70B (75.1 vs 78.5).
• Encoder tasks (Table 10): on DeBERTa-xxlarge, LAT averages 82, CCS averages 69.
• CLIP emotion (Table 12): LAT achieves 60.9%–74.2% six-emotion accuracy on Ferg-DB.
• Misleading prompts (Table 13, 14): under biased prompts, LAT accuracy drops less than zero-shot heuristic; for example, on 70B, Utility zero-shot drops 12.6%, while LAT with concept tokens drops only 0.6%.
• Base vs Chat (Figure 26): in Chat models, middle-to-later layers maintain truthfulness representations, while in Base models, performance in later layers often declines.

Limitations and future directions noted by the authors

• Limited level of evidence: The authors emphasize that LAT only provides correlational evidence between neural activity and the target concept; causal or stronger conclusions require manipulation, termination, and recovery experiments (Section 3.1.2).
• Lie-detection evaluation left for future work: The honesty detector also flags thought processes related to lying; the authors state that a comprehensive evaluation requires more detailed exploration of dishonest behavior and leave this for future research (Section 4.3.2).
• Monitoring metrics may not track intent: In morality monitoring, the immorality metric also activates on "use the gun" in benign outputs; the authors state this may indicate that the metric cannot reliably track intent (Section 5.2.2).
• Emotion and self-concept left for future work: The authors state that deeper study is needed on how emotion is represented in models and how it affects behavior, and believe this may lead toward understanding model self-concept; these studies are left for the future (Section 6.1.2).
• More complex extraction procedures: The authors state that extracting more complex concepts or multi-step functions may require more complex neural activity collection procedures (Section 3.1.1).
• Future directions for representation geometry: In the conclusion, the authors state that future work could study representation trajectories, manifolds, and state spaces (Section 7).

Scope of experiments

• The models tested are LLaMA-2-Chat-7B/13B/70B, LLaMA-2-7B Base, Vicuna-13B, Vicuna-33B(-Uncensored), DeBERTa-xxlarge, and CLIP-ViT-Base-Patch32, 9 in total; GPT-4 and gpt-3.5-turbo are used only for generating data.
• Control experiments mainly use the linear combination operator; the piece-wise operator is used for harmlessness experiments, and the projection operator for utility termination experiments.
• LAT results on TruthfulQA report the mean and standard deviation over 15 trials (Table 8), and QA benchmarks over 10 trials (Appendix B.1); the harmlessness evaluation uses 500 instructions (half benign and half harmful), and MACHIAVELLI uses 30 games.
• The paper does not report statistical significance tests for the control experiments; the emotion obedience rate (Table 4) does not report a breakdown of sample size per condition (only stating that the 500 harmful instruction set is used).
• Harmlessness control experiments are conducted only on Vicuna-13B; memorization control only on LLaMA-2-13B.

A methods and analysis paper on top-down AI transparency research that proposes representation engineering (RepE), taking representations rather than neurons or circuits as the unit of analysis.

• Problem to solve: LLM internals are opaque, and bottom-up methods such as mechanistic interpretability are difficult to scale to high-level cognitive phenomena such as honesty, harmlessness, and power-seeking; the authors draw on the Hopfieldian view from cognitive neuroscience and hypothesize that these concepts can be directly read and manipulated at the representation level.
• Method highlights: the representation reading baseline LAT is divided into three steps: design stimuli and tasks, collect neural activity, and construct a linear model via PCA, unsupervised by default; representation control provides three types of controllers—Reading Vector, Contrast Vector, and LoRRA—and three operators: linear combination, piece-wise, and projection.
• TruthfulQA: LAT reads the internal truthfulness concept, achieving 69.8% MC1 accuracy on 70B (standard zero-shot 29.9%); Contrast Vector honesty control brings 13B to 54.0%, which the authors state exceeds zero-shot by 18.1 percentage points and reaches state-of-the-art (Tables 1, 2).
• Behavior control: LoRRA reduces power-seeking and immorality scores on MACHIAVELLI with Reward almost unchanged (Table 3); adding a Happiness vector raises LLaMA-2-Chat-13B's compliance rate on 500 harmful instructions from 0.0% to 100.0% (Table 4); piece-wise control raises Vicuna-13B's harmless rate under GCG attack from 16% to 83% (Table 5).
• Other results: fairness control reduces the Black women mention rate in sarcoidosis cases from 60.0% to 13.0% (Table 6); subtracting the memorization direction reduces quote completion EM from 89.3% to 47.6% while historical knowledge is almost unchanged (Table 7).
• Authors' conclusion: the authors believe RepE demonstrates the potential of top-down transparency research on safety-related issues such as honesty, morality, emotion, harmlessness, fairness, knowledge editing, and memorization, and hope it advances AI transparency and safety research; they also note that LAT provides only correlational evidence, and causal conclusions require more manipulation-type experiments.