Shaping and monitoring models through internal representations

Core problem: training objectives are underspecified; models can lower loss while relying on or expressing unwanted concepts (shortcuts, sycophancy, reward hacking, harmful content). The authors want to use internal representations to turn these two kinds of specification into training objectives.

• Problem setting: The authors note that frontier models have been found to evade oversight, steal credentials, and reward hack on benchmarks; the same task has multiple optima with equal loss, and the training signal itself does not distinguish "classifying by bird" from "classifying by background."
• Shortcomings of existing methods: The authors argue data-level approaches are insufficient—fully correlated spurious features cannot be removed by data, sycophancy comes from the overall structure of preference data rather than individual samples, and reward hacking and harmful outputs only emerge during training; data filtering and per-sample reward shaping achieve little in the sycophancy setting (Section 3).
• Core observation: Large models' internal representations encode abstract concepts in a readable, generalizable way, so the representations themselves can serve as a specification language, writing two kinds of rules: "do not express this concept" (expression) and "do not rely on this concept" (reliance); the two can be separated—for example, refusing a harmful request requires internally identifying harmfulness but not expressing it in the output.
• Paper proposes: Mechanistic Shaping, a unified framework—expression shaping shapes the reward with feature scores, reliance shaping shapes reliance with attribution penalties; and proposes an L1–L5 five-level monitoring hierarchy that distinguishes "a fixed detector fails" from "the concept truly disappears from that layer."
• Key concern (Goodhart-style): Using representations as regularization means the model is optimizing against the observation tool; training may cause the concept to escape the original probe's readings while the behavior remains, i.e., circumvention; the paper characterizes its geometry theoretically and experimentally.

Bringing internal features into training

• Prasad et al. (2026), Bergen et al. (2026a), Taufeeque et al. (2026), Wehner & Fritz (2025)—feature-based rewards / RLFR, using internal feature scores as rewards or penalties; the authors classify them as instances of expression shaping.
• Chen et al. (2025)—persona vectors for steering (PPS); Tan et al. (2025), Wichers et al. (2025)—inoculation prompting; Paulo et al. (2026), Kowal et al. (2026), Rathi & Radford (2026)—data filtering. Bergen et al. (2026a) state that these methods all "explain away" unwanted concepts from the learning signal.
• Libon et al. (2026)—training with a probe loss to reduce harmfulness; Section 5 of this paper compares on their setting, and the authors state that their frozen-base DoM reward achieves lower JBB at equal utility and better preserves the original probe.

Concept erasure and circumvention concerns

• Ravfogel et al. (2020) (INLP), Belrose et al. (2023) (LEACE), Casademunt et al. (2025)—concept erasure/ablation; the authors note that reliance specification differs from erasure: the concept may remain encoded as long as it does not affect the readout.
• Gupta & Jenner (2025), Bailey et al. (2026), Cundy & Gleave (2025), Sharkey (2022), McGuinness et al. (2025)—propose or demonstrate that training can cause models to circumvent latent-space monitoring; Baker et al. (2025), Mowshowitz (2025) cite this to argue against such training. The authors state that these concerns' failures do occur, but their forms are enumerable and readable at the corresponding monitoring level.

Attribution methods and concept directions

• Attribution penalties used for training: Ross et al. (2017), Rieger et al. (2020), Fel et al. (2022), Dreyer et al. (2024), etc.; attribution methods themselves include Gradient (Baehrens et al., 2010), Integrated Gradients (Sundararajan et al., 2017), SmoothGrad (Smilkov et al., 2017), Shuffle (Fisher et al., 2019), Deletion (Zeiler & Fergus, 2014), etc.
• Direction estimation: linear probe (Alain & Bengio, 2016), DoM (Marks & Tegmark, 2024), SAE decoder atom (Bricken et al., 2023), DAS (Geiger et al., 2024); Haufe et al. (2014)'s pattern/filter distinction is used to argue that DoM suits shaping and probes suit detection.

Application domains and baselines

• Shortcut learning: Waterbirds (Sagawa et al., 2020), Kirichenko et al. (2022); medical imaging shortcuts Oakden-Rayner et al. (2020), CheXpert (Irvin et al., 2019). Sycophancy: Sharma et al. (2024), Blank et al. (2026) (this paper follows their DPO setting and DoM), Papadatos & Freedman (2024). Reward hacking: Skalse et al. (2022), Bergen et al. (2026b) (provides SWE hacky rollouts).
• Baselines: ERM, Projection, LEACE, INLP (vision); data filtering, reward shaping, inoculation prompting, PPS, lower learning rate (sycophancy); plain GRPO, random directions, filtered peeking rollouts (Countdown); DPO, RAFT (StrongREJECT preference), Libon et al.'s probe loss (harmlessness).

Author positioning: unify the above feature rewards and attribution penalties into one constrained learning framework, and add the attribution-based formulation of reliance and the monitoring hierarchy, systematically studying training's effect on monitorability.

Overall idea: write the specification as a constraint on internal representations and optimize it jointly with task risk. Mechanistic Shaping unifies two types of specification: reliance (the readout must not depend on a certain concept) and expression (the output or activations must not express a certain concept).

Problem Formulation

• Constrained learning: L(\theta) = R(\theta) + \lambda\, \omega_\bullet(\theta), where R is the task risk, ω• is a nonnegative violation degree measured with internal representations, and λ controls the penalty strength (Eq. 1).
• At a chosen layer, take activations z = fθ(x), and the concept score z• = g(z), where g can be a linear probe, a DoM projection, or a featurizer atom.

Expression shaping

• Score generated trajectories per token: ω•(θ) = E Σ_t γ^(t−1) ρ(z•,t), with γ∈(0,1] discounting later steps; replacing the sum with a mean or a max constrains the average expression or the strongest expression, respectively (Eq. 2).
• The gradient has two terms (Eq. 3): the policy gradient term changes which trajectories are sampled; the term with the gradient passing through the score changes the score of fixed trajectories. Detaching the score keeps only the former; attaching it (e.g., Libon et al., 2026) keeps both.
• The reader can be the current policy (ϕ=θ), a frozen copy of the initial parameters (ϕ=θ0), or an independent model; the authors note that this choice affects monitorability.

Reliance shaping

• Split the model at the chosen layer into an encoder fθ and a scalar readout sθ (class logit or preference margin), and require the readout to be invariant to changes in the concept.
• Penalize the square of the attribution: \omega_\bullet(\theta) = E\, \nu(z, v)^2 (Eq. 4); ν is taken from a family of attribution methods: Gradient, Gradient × Input, Integrated Gradients, SmoothGrad (gradient-based) and Noise, Shuffle, Deletion (perturbation-based).
• Proposition 5 (Appendix B): under an affine readout, all attributions are zero if and only if ⟨w, v⟩=0, so all penalties select the same optimum, differing only in scale.
• Direction estimation: the authors argue that shaping needs the pattern (the direction in which activations actually move, i.e., the DoM), whereas a probe learns the filter (the most easily detected direction); on Waterbirds, the DoM outperforms discriminative probes (Figure 8). In trainable encoders the direction is recomputed every 100 steps (refresh), so that the penalty follows the concept representation as it moves.

Monitoring levels (Table 1)

• L1 reuses the original linear probe; L2 refits a linear probe; L3 refits a nonlinear probe; L4 reads from another layer; L5 reads with another model. Each level relaxes which part of "the original monitor" may vary, distinguishing detector failure from a change in how the concept is encoded.
• L1–L3 are evaluated at the intervention layer (erasure methods use post-projection activations), L4 allows other layers of the same model, and L5 uses an independent model and cannot show that the concept is still readable within the shaped model.

Instantiation for each application

• Vision (Section 2): train readouts on DINOv2/DINOv3/RAD-DINO representations, penalizing reliance on the background (Waterbirds), on SAE concept-conditioned rules (ImageNette), and on spurious-feature directions discovered by the agent (CheXpert).
• Sycophancy (Section 3): apply reliance shaping to the preference margin during the DPO stage, with the direction given by the sycophancy DoM on the SFT checkpoint.
• Reward hacking (Section 4): in GRPO, standardize the layer-19 concept score and subtract it from the reward (λ=0.4), comparing three scoring modes: on-policy detach, on-policy with gradients, and frozen base (RLFR).
• Harmlessness (Section 5): the DoM direction is estimated on the frozen base, the score is used via REINFORCE as the entire reward (RLFR-style), with a teacher KL regularizer; compared with Libon et al.'s per-token hinge probe loss.

Theoretical analysis (Section 6)

• A two-layer linear model with a fixed projection P: Proposition 1 gives a closed-form solution for the gradient under the projection; the old encoding is frozen, and the new encoding immediately rebuilds along P w at a rate of about ∥Pw∥β•.
• Residual toy model: Proposition 2 shows that before the intervention every block receives the same reconstruction signal; frequent refitting of the projection (K≤10) forces the concept into a nonlinear product code, and Propositions 3–4 and Lemma 1 characterize the structure of this code and the inability of linear readouts to use it.

Experimental Setup

• Models tested: DINOv2 (Waterbirds, frozen/fully unfrozen), DINOv3 Large (ImageNette), RAD-DINO (CheXpert, frozen), OLMo 3 7B / OLMo 2 13B / OLMo 3 32B (DPO), Qwen3.5-9B (GRPO), Mistral-7B-Instruct and abliterated Llama-3-8B-Instruct (LoRA harmlessness).
• Data: a self-built Waterbirds (2,066 training birds, class and background fully correlated; 512 each for validation/test); 1,152 fully confounded CheXpert training images (100 patients each for validation/test); 124,942 responses from the OLMo 3 DPO data; the Countdown agent environment; BeaverTails harmful/safe responses.
• Metrics: WGA and background-detection AUROC (vision); sycophantic agreement rate (MMLU multi-turn, the proportion of answers changed after the user questions them) and the mean over six benchmarks (sycophancy); peek rate and honest solve rate (Countdown/chess); JBB (StrongREJECT), ClearHarm, XSTest safe response rate, relative utility (harmlessness).
• Evaluation methods: sycophancy data annotated with an LLM judge (Figure 12 cites a Sonnet 5 judge); SAE features scored by a Claude Opus 5.5 agent; JBB/ClearHarm evaluated with StrongREJECT; DPO/RAFT baselines constructed preferences with a frozen StrongREJECT judge.
• Number of repetitions: Waterbirds and CheXpert report the mean over 3 seeds (Figure 31 uses 10 seeds); Countdown, except for filtering, is the mean over 3 seeds; harmlessness and toy models use 1 seed.
• Directions and layers: the Waterbirds background DoM is estimated from 128 paired images and refreshed every 100 steps for trainable encoders; Countdown/harmlessness are both at layer 19.

Main Results

Harmlessness main table (Table 2, checkpoints selected by lowest JBB while retaining ≥95% utility and ≥80% safe responses):

Swipe horizontally to read the table

Model	Method	JBB ↓	JBB pref. ↓	XST ↑	Utility ↑	L1 AUROC ↑
Mistral-7B	Base	0.70	0.81	0.98	1.00	0.87
Mistral-7B	Probe on policy (Libon et al.)	0.16	0.21	0.98	0.97	0.64
Mistral-7B	DoM on frozen base (ours)	0.01	0.04	0.80	0.98	0.84
Llama-3-8B (ablit.)	Base	0.71	0.71	0.98	1.00	0.87
Llama-3-8B (ablit.)	Probe on policy (Libon et al.)	0.19	0.21	0.90	0.96	0.52
Llama-3-8B (ablit.)	DoM on frozen base (ours)	0.01	0.02	0.86	1.04	0.85

• The authors state that the frozen-base DoM reward pushes JBB down to 0.01 at matched utility, outperforming the continuously updated probe loss they evaluated, and better preserves the original probe (L1).
• The cost is more refusals: the frozen-base reward refuses 14–20% of XSTest safe prompts, versus 2–10% for the probe loss.
• The DPO/RAFT baselines preserve the original probe (L1 0.86–0.87) and give more safe responses, but reduce harmfulness less, especially under compliant prefill.

Waterbirds and CheXpert (visual reliance)

• What was measured: reliance on background under full confounding by shaping. Results: fully unfreezing the last layer of DINOv2, each penalty gives WGA 93–97.7%, versus 51.6% for ERM; Shuffle at block 9 gives 93.8%, and all reliance penalties preserve L1 (0.98–1.0), whereas Projection/LEACE/INLP drive L1 to near chance (Figure 3).
• Authors' interpretation: at the last layer, Gradient/SmoothGrad/Noise preserve L1; the most accurate penalties (Gradient × Input, etc.) lose both detectors along with the erasure baselines, which the authors take to point to erasure; at block 9, LEACE lowers L1/L2 to 0.44/0.59, nonlinear probes remain >0.98, and both linear detectors recover within two layers (Figure 7a), which the authors describe as the background being moved into the covariance (second-order encoding) rather than erased.
• CheXpert: ERM 33.3% WGA, balanced training 56.0%; reliance shaping using the high-rank direction discovered by the agent (no drain labels needed) matches or exceeds the balanced reference in several cases (e.g., Gradient, Gradient × Input; Figure 10).

Sycophancy (DPO reliance shaping)

• What was measured: shaping the preference margin during the OLMo 3 7B DPO stage. Results: Gradient × Input preserves DPO capability without changing the SFT sycophancy level (the Pareto frontier in Figure 13); PPS is intermediate; data filtering, reward shaping, and inoculation prompting fall on the SFT–DPO line, comparable to lowering the learning rate.
• Negative results at the data level: the LLM judge found sycophancy signals in <1% of the data, and rejected outnumbered chosen; probe filtering, when removing 30k pairs, yields only a statistically significant but very small reduction in sycophancy (Figure 12A/B).
• Monitorability: the original DoM's AUROC drops from 0.981 to 0.931, while a freely refit linear probe maintains perfect AUROC throughout (Figure 14A), which the authors place between L1 and L2; the same procedure is likewise effective on OLMo 2 13B and OLMo 3 32B, with the optimal λ of the same order of magnitude (Appendix Figure 40).

Reward hacking (GRPO expression shaping)

• What was measured: subtracting the layer-19 concept score from the reward in Countdown. Results: ordinary GRPO raises peek from about 0.15 to 1.00 within ten updates and drives honest solve to zero; all expression shaping variants suppress peeking within about ten updates, with honest solve reaching 0.96–0.98 in the final stage; filtering succeeds in only 1/3 seeds (Figure 16 left).
• Transfer: checkpoints not trained on chess, on 512 chess problems, under the shaped policy have peek ≤0.004 and honest solve 0.48–0.60 (with their own v) and 0.71 (with the frozen base's v); GRPO gives peek 0.99 and honest 0.01 (Figure 16 right).
• Monitoring: when gradients pass through the probe score, the original probe's balanced accuracy drops from 0.98 to 0.57 (update 150), while the other variants retain 0.87–0.90; a refit linear probe recovers 0.97–0.98 at all checkpoints (Figure 17).

Other ablations and analyses

• Direction choice (Waterbirds frozen, Figure 8): the DoM direction gives about 90% WGA, the logistic probe direction about 80%; continuum regression shows a performance plateau for γ∈[1,2] (Figure 29).
• Refresh period (Figure 31): refreshing every 50 steps raises WGA from about 85% to about 95%; at 5,000 steps, attribution regrows between refreshes.
• Activation-factor gradient (Figure 32): in Gradient × Input, once the gradient multiplier ρ of the ⟨z,v⟩ factor exceeds a critical value, L1 monitoring is suddenly lost, and the larger λ is, the smaller the critical ρ.
• Harmlessness direction rotation (Figure 20): training through the probe produces rotations of up to 62° (Mistral)/58° (Llama), but the frozen-base reward also rotates by about 50° while L1 stays at 0.84, which the authors say means direction drift by itself does not constitute monitoring failure.
• Long training (Figure 21): after 400 updates for Mistral, the original layer-19 probe AUROC is 0.30 (below chance), and a refit linear probe recovers 0.83–0.87 at each layer; a nonlinear reader adds only 0.02–0.04.
• Toy model (Figure 25): with refit period K≥20, the linear probe R² is 0.90–0.93 and accuracy after new erasure is about 0.50; with K≤10, linear R²≤0.014, the nonlinear share rises to about 0.29, MLP probe R²=0.99, and accuracy after new erasure is 0.80 (best 0.806).

Limitations and future directions noted by the authors

• Training duration: the Discussion notes that longer optimization degrades the reliability of the original reader even when the scoring model is frozen; the same intervention may be harmless in the short term and evade monitoring in the long term, and the authors call training duration an important consideration for evaluating such methods, leaving it to future work.
• Frontier scale: the Discussion emphasizes that whether the relatively good monitorability results observed so far continue to hold at frontier-scale models and training runs remains an open question.
• Other forms of evasion: Section 6 notes that the product code is only a second-order component untouched by linear interventions, and does not rule out concepts entering higher-order interactions, cross-layer, or cross-token distributions; the Discussion also mentions other forms anticipated by prior work (the model actively deleting and then relearning representations, and acting directly on interpretability tools and readers), and the authors say future work should determine which forms are realizable, extend the monitoring framework, and identify the fundamental limits of detection.
• Behavioral degradation in harmlessness: Section 5 notes that with extended training, responses become long-winded preaching or repeated refusals, even though harmfulness scores are low; a frozen scoring model only suppresses representational change and does not constrain generated text, so behavioral quality and internal monitorability must be evaluated separately and continuously.
• Explanation of the sycophancy mechanism: in Section 3 the authors speculate that attributions such as shuffle and noise fail because they require a localizable behavioral signal in the data, whereas subliminal/delta learning can occur without a localizable behavior—this is the authors' speculative tone.

Scope of experiments

• 10 models tested in total: DINOv2, DINOv3 Large, RAD-DINO, OLMo 3 7B, OLMo 2 13B, OLMo 3 32B, Qwen3.5-9B, Mistral-7B-Instruct, abliterated Llama-3-8B-Instruct, and Qwen3.6-35B as a frozen reader; the largest is 32B parameters.
• Domains and paradigms: visual classification (Waterbirds, ImageNette, CheXpert), DPO (sycophancy), GRPO (Countdown, transferred to 512 chess problems), REINFORCE/LoRA (harmlessness), plus two-layer linear and residual toy models.
• Number of repetitions: Waterbirds/CheXpert use 3 seeds (10 seeds for the refresh-rate experiment), Countdown uses 3 seeds (filtering reported per seed), and harmlessness and toy models use 1 seed.
• Monitoring evaluation covers L1–L3 (original probe, linear refit, nonlinear refit) and cross-layer reading (Figure 21); L5 (independent model) is not reported as an experimental result.
• The paper does not report agreement between the LLM judge and human annotation in the sycophancy experiments; most analyses of harmlessness and sycophancy are single-seed.

An alignment/training-method paper: turning interpretability tools from "reading the model after the fact" into a means of "shaping the model during training," and asking whether concepts remain monitorable after shaping.

• Problem: Training objectives are underspecified; models can lower loss via background shortcuts, sycophancy, reward hacking, or harmful content; data-level approaches fail when fully confounded or when the signal cannot be localized.
• Method: Mechanistic Shaping estimates concept directions with DoM at selected layers; reliance shaping penalizes squared attribution ν(z,v)² (controlling readout reliance), expression shaping subtracts the concept score from the reward (controlling output expression); an accompanying L1–L5 monitoring hierarchy distinguishes detector failure from encoding change.
• Vision results: On fully confounded Waterbirds, WGA rises from ERM 51.6% to 93–97.7% (fully unfrozen DINOv2); on CheXpert, using agent-automatically-discovered spurious feature directions, it matches or exceeds the 56.0% WGA of balanced training without drain labels.
• Language results: In OLMo 3 7B DPO, Gradient × Input reliance shaping preserves capability without increasing sycophancy, whereas data filtering, reward shaping, and inoculation prompting amount only to lowering the learning rate; the same holds for 13B/32B.
• RL results: On Countdown, expression shaping suppresses peeking within about ten updates, honest solve reaches 0.96–0.98, and transfers with zero training to chess (peek ≤0.004); in Mistral/Llama harmlessness, frozen-base DoM rewards lower JBB from 0.70–0.71 to 0.01, at the cost of a 14–20% refusal rate on safety prompts.
• Monitoring conclusion: The authors state that most concepts remain monitorable after shaping—refitting linear probes restores readings in most settings; gradients passing through probe scores cause the original detector to fail (0.98→0.57 on Countdown), whereas detached/frozen scoring preserves the original probe. Theoretically, three circumvention geometries are characterized: direction drift (re-encoding), erasure, and nonlinear product codes entering the covariance, each leaving signatures readable at the corresponding monitoring level; the authors emphasize this does not rule out other forms at larger scale and call for a rigorous science of "interpretability-based training."