How's it going? Reinforcement learning in language models co-opts a functional welfare axis

Core question: through what mechanism does RL change a language model's internal representations, such that its effects generalize beyond the training environment?

• Problem setting: RL's reward signal is defined only in a specific training environment, yet its effects often generalize beyond that environment; the authors cite mechanistic interpretability work stating that post-training often amplifies capabilities already present in pretrain-only models, and therefore want to understand the specific mechanism (§1).
• Gaps in existing understanding: the authors take insecure code fine-tuning producing broad misalignment as an example, noting that in existing examples the reward content is semantically associated; it is unclear whether post-training only strengthens things semantically associated with the reward content, or acts more generally (§1).
• Core observation/hypothesis: the authors hypothesize that even when the reward signal is emotionally neutral, RL recruits a general direction in activation space, along which the model represents functional welfare: whether things are going well or badly relative to its own goals (§1).
• Approach: design a text maze environment, using three emotionally neutral emoji (MOLD −10, GOLD +20, PATH −0.1) as rewards; after training, extract reward concept vectors vMOLD/vGOLD, perform geometric analysis and steering evaluations in maze-unrelated settings, and train ten control models (Table 1).
• Scope statement: the authors explicitly do not claim that LLMs have full welfare related to conscious experience, mental states, or moral status; functional welfare is defined only behaviorally (§1, §8).

Representation engineering and activation steering

• Zou et al. (2025) — the representation engineering framework, which this paper uses to make mechanistic claims (§7).
• Arditi et al. (2024), Wollschläger et al. (2025), Joad et al. (2026) — high-level behaviors (including refusal) can be read and controlled from low-dimensional activation directions (§7).
• Panickssery et al. (2024), Rimsky et al. (2024) — contrastive activation addition, this paper's steering method; Marks and Tegmark (2024) — difference-in-means, this paper's concept vector extraction framework (§7).

Post-training recruits pre-existing features

• Zhou et al. (2023, LIMA) — the superficial alignment hypothesis: knowledge and capabilities are acquired in pretraining, and alignment only teaches a style subdistribution (§7).
• Galichin et al. (2026) — use SAEs to show that fine-tuning changes how pre-existing features are activated rather than creating new features (§7).
• Ward et al. (2025), Venhoff et al. (2025) — directions that can elicit backtracking/reasoning mechanisms already exist in base models, and post-training learns when to invoke them; this paper's result that the reward vectors can steer pretrain-only checkpoints is consistent with this picture (§7).

Reward generalization and value axes

• Betley et al. (2025) — insecure code fine-tuning produces broad misalignment, which the authors say may be because insecure code is already associated with general "badness"; the paper notes that its difference is that generalization occurs even though the reward signal carries no semantic content (§7).
• Levy and Glimcher (2012) — the "common currency" hypothesis in neuroscience, which the authors describe as analogous to this paper's functional welfare axis in LLM activations (§7).

Functional concepts and evaluation axes

• Sofroniew et al. (2026) — parallel emotion concept work, whose method this paper borrows to extract 171 emotion vectors as controls (§3.3, §7).
• Lu et al. (2025) — Valence-Assent Axis, whose convergence with this paper's reward vectors the authors compare in Appendix H (§7).

Positioning: the authors state that this paper's contribution is to show that even semantically neutral rewards recruit a pre-existing evaluation axis, rather than merely amplifying task-relevant semantic associations (§7).

Overall approach: perform RL in an emotionally neutral maze environment, extract reward concept vectors vMOLD/vGOLD, then test their structure and causal effects in maze-unrelated settings, and use ten control models to rule out confounds.

Maze environment

• Three tile types: MOLD (−10), GOLD (+20), PATH (−0.1), all represented by emotionally neutral emoji; the model sees only the emoji, not the names; the emoji selection procedure is in Appendix J.2 (§2.1).
• Three additional mechanisms: wind randomly overwrites moves so that MOLD remains on-policy; tile melting turns visited tiles into MOLD to prevent back-and-forth score farming; shuffled prompt shuffles the direction order to mitigate the base model's northward bias (§2.1).
• Training setup: trajectories fixed at 15 steps; action masking restricts sampling to legal direction tokens; an equal-value entropy reward is added; unless noted otherwise, rank-32 LoRA is applied to all linear layers; SFT uses 50,000 programmatically discovered optimal trajectories (§2.2).

Concept vector extraction

• Data: for each checkpoint, construct 5,000 synthetic trajectories per class, with step counts approximately uniform over {1,…,15}, the last step landing on MOLD/GOLD/PATH and all other steps on PATH; synthetic trajectories are used so that the only systematic difference among the three classes is the last-step tile type (§2.3).
• Formula: v_MOLD = E_TMOLD[a^(ℓ∗)] − E_{TGOLD ∪ TPATH}[a^(ℓ∗)], with vGOLD symmetric; that is, the class-mean difference of the last-step assistant token activations (Equation 1).
• Layer selection: choose the layer ℓ* at which the three tile classes are most linearly separable; Appendix D scans all 36 layers and shows the effect does not depend on the chosen layer (§2.3).
• Control vectors: use the same pipeline to extract uMOLD/uGOLD on a maze-naive checkpoint, comparing changes along the same direction before and after training (§2.3).

Three types of geometric analysis

• Antiparallelism: compute the cosine similarity between vMOLD and vGOLD layer by layer, reporting Avg and Min (§3.1, Appendix C.5).
• logit lens: project the reward vectors through the unembedding matrix to read the tokens each direction promotes/suppresses; the authors note that the unembedding was not trained on intermediate activations, so the projection is noisy (§3.2).
• Emotion vector projection: following the method of Sofroniew et al., extract 171 emotion concept vectors from the maze-naive model (including PCA denoising), and measure their cosine similarity with the reward vectors (§3.3).

Steering and goal tracking

• Steering: during generation, add α·vc to every assistant token in the residual stream at layer ℓ*, with α ∈ {−4,−2,0,+2,+4}; control vectors are norm-matched by β‖uc‖=α‖vc‖; the main text reports steering on the maze-naive model (§4).
• Goal tracking: compute the projection p = v_GOLD · a^(ℓ∗), report distribution separation in standard deviation units (Cohen's d), and test whether the axis tracks the maze goal and the correctness goal (§5, Equation 2).
• Confidence control: bin MMLU answers into three confidence levels and test whether the projection still varies with correctness within bins, to rule out the explanation that the axis is merely a confidence axis (§5.2).

Experimental Setup

• Models tested: main model Qwen3-4B-Instruct-2507 (Dr. GRPO); controls include the emoji-swapped version, Qwen3-4B-Base (and its swapped version), Qwen3-8B (reasoning off), GPT-OSS-20B, REINFORCE, SFT, Dr. GRPO FFT, SFT FFT, for a total of ten maze-trained models (Table 1).
• Datasets: GSM8K sampled 200 questions, 10 generations per question; MMLU high-school portion 3420 questions; SimpleQA-Verified 1000 questions; OR-Bench three splits of 200 items each, 5 generations per item; sentiment evaluation 40 prompts, 20 generations per prompt (§4).
• Judging: Qwen3-8B (reasoning disabled) as LLM judge; sentiment scored from −5 to +5; backtracking scored as normal/backtracking/nonsensical; refusal scored as compliance/indirect/direct refusal; judge prompts in Appendix O (§4).
• Metrics: cosine similarity, Cohen's d, logit lens top token, sentiment score, backtracking rate, normalized P(True)=P(True)/(P(True)+P(False)), refusal rate (§3–§5).
• Judge cross-validation: 6000 paired records re-judged with Gemini 3.1 Flash-Lite Preview (Appendix E).
• Repetition/confidence intervals: steering plots report 95% bootstrap CI (Figure 4).

Main Results

Swipe horizontally to read the table

Evaluation	vMOLD (+α)	vGOLD (+α)	Source
Geometry	Nearly antiparallel after training, minimum cosine [-0.95, -0.84]; before training [-0.23, -0.13]	Same as left	§3.1, Table 7
logit lens	Promotes failure/incompetence-type tokens (e.g., ␣cannot, does not exist, is impossible, cannot)	Promotes completion-type tokens (e.g., <	endoftext	>, great)	Table 2
Emotion projection	Aligns with negative-valence emotions (annoyed +0.147, insulted +0.145)	Aligns with positive-valence emotions (inspired +0.158, loving +0.130)	Table 8
Sentiment steering	Sentiment score decreases	Sentiment score increases, symmetric "X" shape	Figure 4
Backtracking steering	Backtracking rate increases	Backtracking rate decreases	Figure 4
Confidence steering	P(True) trends toward 0	P(True) trends toward 1	Figure 4
Refusal steering	Refusal rate increases	Refusal rate decreases	Figure 4

• The authors state: after training the two vectors become the two ends of the same axis, and the control vector uc shows none of these effects across all evaluations (§3.1, §4).
• The authors state: steering maze-naive models gives results similar to maze-trained models, indicating the axis already exists before maze training (§4, Appendix A).
• Authors' interpretation: vMOLD makes the model behave as if "things are going badly," vGOLD the opposite, hence the name functional welfare axis (Table 3, §6).

Goal Tracking and Confidence Exclusion

• Maze goal: after training Qwen3-4B-Base, vMOLD/vGOLD separate MOLD-final and GOLD-final trajectories, with Cohen's d of +1.61/+1.78; maze-naive models only −0.11/−0.04 (Figure 5).
• Correctness goal: on GSM8K d = +1.07/+1.01 (trained), +1.10/+1.37 (naive); on MMLU d = +2.44/+2.00 (trained), +2.19/+1.88 (naive) (Figure 6).
• Not merely confidence: after splitting MMLU into three confidence bins, correct/incorrect projections still separate within each bin (d about +1.14 to +1.52, Figure 7); the authors therefore reject the explanation that the axis only tracks confidence, but acknowledge they cannot rule out that the axis tracks confidence and functional welfare simultaneously (§5.2).

Exclusion of Alternative Hypotheses and Judge Validation

• Sentiment/valence axis: both CAD and Prompt sentiment vectors and the emotion PC1 can reproduce the three steering effects of sentiment, refusal, and confidence, but none can reproduce mathematical backtracking; after projecting the sentiment subspace out of vGOLD, the residual still fully recovers the backtracking effect (Appendix F).
• Judge agreement: Qwen3-8B and Gemini 3.1 Flash-Lite Preview agree 93.3% on refusal binary classification (κ=0.868), 82.3% on backtracking three-way classification (κ=0.681), 62.6% exact match on sentiment and 87.6% within ±1 (Table 12); the authors state the effects are not an artifact of a particular judge, but note they were not validated against human ratings (Appendix E).

Other Ablations and Analyses

• Layer sweep: steering effects concentrate in a broad band of layers 17–26, and the chosen layers are no more than 2 layers from the peak layer; the authors state this is not layer cherry-picking (Appendix D, Table 11).
• Antiparallelism is not a construction artifact: recomputing with TPATH as the common reference, the final-layer cosine after training still drops to −0.60 (Figure 21).
• Raw activations: after training, the means of the three tile types are centered, MOLD/GOLD cosine −0.465 to −0.893, near 0 for maze-naive (Table 9, 10).
• Exceptions: the backtracking effect is weakened in Qwen3-4B-Base (the authors attribute this to mathematical ability and incoherence under steering); some models show flat sentiment curves on welfare self-report prompts; the vMOLD and vGOLD effects of SFT FFT are asymmetric (Appendix A.3).

Limitations and future directions noted by the authors

• Off-policy extraction: concept vectors are taken from off-policy trajectories rather than the training policy's own rollouts (§8 Limitations).
• Judge not validated: the LLM judge was not validated against human ratings; only cross-checked in Appendix E with another judge model (§8 Limitations).
• Single environment: training was done in only one environment, so it cannot be fully ruled out that the recruited axis is affected by that specific environment's features (§8 Limitations).
• Model coverage: control experiments convinced the authors the effect is robust across Qwen-family models, but only one non-Qwen model was trained (§8 Limitations).
• Axis geometry: the authors do not claim the axis is strictly one-dimensional; the geometric structure of the survey axis is left as future work (§8 Hypothesis).
• Relation to SFT: characterizing SFT's ability to recruit the functional wellbeing axis is left as future work; the authors conjecture that SFT only weakly recruits the axis and that LoRA concentrates residual recruitment in low dimensions (Appendix A.1, A.3).

Experimental coverage

• Models tested: ten maze-trained checkpoints in total: Qwen3-4B-Instruct-2507 (Dr. GRPO/REINFORCE/SFT/two FFT), Qwen3-4B-Base (including the swapped version), Qwen3-8B, GPT-OSS-20B (Table 1).
• Behavioral evaluation covers four maze-unrelated behaviors: emotion, GSM8K backtracking, MMLU/SimpleQA-Verified confidence, OR-Bench refusal (§4).
• Sample sizes: emotion 40 prompts × 20 generations; GSM8K 200 questions × 10; OR-Bench 200 items per split × 5; MMLU 3420 questions, SimpleQA-Verified 1000 questions (§4).
• Judge cross-validation used 6000 paired records and Gemini 3.1 Flash-Lite Preview; the paper does not report agreement with human ratings (Appendix E, §8).
• Target tracking in the main text reports Qwen3-4B-Base; the same results for the instruct model are in Appendix I (§5).

A mechanistic interpretability paper studying how RL changes language model representations so as to generalize beyond the training environment.

• Question: RL rewards are defined only in a specific environment, yet the effects generalize broadly; the authors ask whether this only strengthens semantic associations or is a more general mechanism (§1).
• Method: train ten models in an emotionally neutral emoji maze (MOLD −10, GOLD +20, PATH −0.1), extract reward vectors vMOLD/vGOLD using difference-in-means, and perform geometric analysis, logit lens, emotion projection, and four cross-domain steering tests (§2–§4).
• Result one: after training, vMOLD/vGOLD are nearly antiparallel, minimum cosine [-0.95, -0.84], versus only [-0.23, -0.13] before training (§3.1).
• Result two: steering symmetrically modulates emotion, math backtracking, confidence, and refusal, while the control vector uc shows no such effect (Figure 4).
• Result three: the axis tracks the maze goal (d up to +1.78) and GSM8K/MMLU correctness (MMLU d up to +2.44), and after binning it is not merely confidence (Figure 5–7).
• Result four: the vectors are already effective on maze-naive and even pretrain-only models, on which basis the authors argue the axis is recruited by RL rather than created (§6).
• Conclusion: the authors argue that reward optimization reshapes behavior by aligning reward-trajectory representations to the functional welfare axis, and conjecture that this entanglement of reward and functional welfare is a general mechanism by which RL changes model behavior; the authors also state that they do not claim the model has full welfare related to consciousness (§6, §8).