A Bag of Concepts? Testing Role-Filler Bindings in the Verbalizable Workspace
A Bag of Concepts? Testing Role-Filler Binding in the Verbalizable Workspace
The authors test whether the verbalizable workspace carries role-filler binding in three models, and the authors state that the workspace carries only role readout directions rather than the substrate of binding.
- A runtime alignment monitor based on the verbalizable workspace needs the workspace to carry role-filler binding in order to distinguish safety-relevant propositions that share a concept set; Gurnee et al. left open the question of whether the workspace carries binding.
- On Qwen3.6-27B, Gemma-3-12B, and Gemma-3-27B-IT, using two lenses, J-lens and R-lens, and two direction estimators, diff-of-means and LRE, we run three sets of experiments: linear probe decoding, direction intervention (binding score), and ablation, together with strength-matched controls such as random subspaces.
- Residual-stream role decoding is near ceiling (e.g., 0.989 at sentence-final position in Qwen3.6-27B), while J-space is below a same-rank random subspace; LRE directions can shift role readout but the binding score is indistinguishable from controls; ablation impairs binding only at entity tokens in Gemma-3-27B-IT (+0.275).
- The authors note: the identity-swap strength check fails on all three models, limiting the interpretability of the null steering results; the positive ablation in Gemma-3-27B-IT is confounded between instruction tuning and model scale; only three dense transformers, English agent-patient stimuli, and a single layer per model were tested.
- Evaluated models
- Qwen3.6-27BGemma-3-12BGemma-3-27B-IT
- Benchmarks
- 自建角色-填充绑定刺激集(3 或 6 个概念对 × 4 种句法结构,每模型 600 族 2400 句)
- Metrics
- role decoding accuracybinding scorebinding-specific deficitCohen's d
The paper tests whether the verbalizable workspace of language models carries role-filler bindings, finding that the workspace carries the model's readout of roles but not the representations used for its computational binding. The study conducts decoding, intervention and ablation experiments on three models, Qwen3.6-27B, Gemma-3-12B and Gemma-3-27B-IT, using two lenses, J-lens and R-lens, and two direction estimators, mean difference and LRE gradient. Linear probes decode roles from the residual stream near ceiling (0.975–0.989), but only 0.57–0.59 from J-space, below same-rank random controls; interventions change the model's reports of roles in two of the three models, but binding scores cannot be distinguished from intensity-matched controls. Ablating J-space causes specific binding damage only on entity tokens in Gemma-3-27B-IT (+0.275), replicated across three lens variants, and the authors note this result is confounded by instruction tuning and model scale. The authors accordingly argue that runtime alignment monitoring based on workspace readout will give similar responses on relational propositions where benign and problematic states share a concept set.
Why it was selectedThe paper uses decoding, intervention and ablation to test whether the verbalizable workspace carries role-filler bindings, delineating the capability boundary for runtime alignment monitoring that relies on workspace readout.
Paper analysis
What problem does this paper attempt to solve?
Whether the verbalizable workspace carries role-filler binding determines whether an alignment monitor based on it can distinguish propositions that share a concept set.
Core question: does the verbalizable workspace (J-space/R-space) carry role-filler binding, or does it carry only a bag of concepts.
- Monitoring scenario: Gurnee et al. [2026] propose a small subspace in the residual stream that can be projected into vocabulary space and read as natural language, which can serve as a substrate for runtime alignment monitoring, surfacing the model's silent cognition before it outputs.
- Why binding matters: the authors argue that most safety-relevant propositions are relational—"the model plans to deceive the user" and "the user plans to deceive the model" share the concept set {model, user, plan, deceive} and differ only in role assignment; a monitor that reads only co-occurring concepts and not binding cannot distinguish the two.
- Open gap: Gurnee et al. [2026] explicitly acknowledge that J-lens readout is a set rather than a structure, and leave "whether concepts are bound to semantic roles" as an open question; the R-lens of Blank et al. [2026] uses layer-wise relevance propagation to index directions by relation, and the authors state that its authors report its directions are more causally loaded than J-lens.
- Core distinction: the authors distinguish readout (the model's verbal report about a role) from computation (the representation the model uses internally to assign roles); the workspace may carry only the direction that steers the former without hosting the substrate of the latter.
- Paper's approach: across three models (Qwen3.6-27B, Gemma-3-12B, Gemma-3-27B-IT), two lenses (J-lens, R-lens), and two role-direction estimators (diff-of-means, LRE-gradient), test binding with three sets of experiments—decoding, intervention, and ablation—and use internal positive controls to distinguish "truly no binding" from "weak estimator/tool artifact/model artifact".
What related research exists?
Related work falls into four groups: J-lens and the workspace, R-lens, binding in language models, and causal direction estimators; the closest is Feng et al. [2025].
J-lens and workspace
- logit lens [Nostalgebraist, 2020]: projects intermediate-layer activations through the unembedding matrix to see which tokens each layer favors.
- tuned lens [Belrose et al., 2023]: learns an affine transformation to map intermediate-layer activations into the final-layer coordinate system, without needing the full Jacobian.
- Sparse autoencoders [Bricken et al., 2023]: decompose activations into more interpretable features, at the cost that each concept requires manual identification of features.
- J-lens [Gurnee et al., 2026]: identifies a verbalizable J-space in the residual stream, where each vocabulary token is associated with a direction that on average causally boosts its logit; the authors relay that it accounts for 6–7% of activation variance, is causally decisive on concept-swap interventions, and its authors acknowledge that the readout is a set rather than a structure.
R-lens
- R-lens [Blank et al., 2026]: a structural extension of the same paradigm, using layer-wise relevance propagation to index workspace directions by relations rather than vocabulary tokens; this paper compares R-lens with the J-lens released alongside it and fitted on the same forward passes and corpus, so that differences between R and J can be attributed to the lens type.
Binding in language models
- Feng and Steinhardt [2024]: propose binding IDs and show that role-filler information can be recovered from the full residual stream.
- Feng et al. [2025]: propose propositional probes, discovering subspaces containing propositional information from activations post hoc; the paper states that this is the closest work to this paper, with the difference being that this paper asks whether binding already exists in a pre-identified workspace (J-space), rather than discovering binding subspaces from the residual stream.
- Greff et al. [2020]: provides a theoretical basis for distinguishing bag-of-concepts from relational representations.
Causal direction estimators
- Hewitt and Liang [2019], Pimentel et al. [2020]: probe accuracy may reflect probe expressivity rather than representational structure.
- Chanin et al. [2024]: propose LRC and LRE, constructing interpretable concept directions from relations between representations as a more principled alternative to simple class-mean differences; this paper's LRE gradient estimator is adapted from their concept directions.
Positioning: the authors state that this paper does not discover binding subspaces post hoc, but directly tests the open question left by Gurnee et al. [2026]; this paper has no behavioral baseline methods, and its controls are internal controls such as same-rank random subspaces, random directions, and shuffled-label directions.
How does the paper address this problem?
Three sets of experiments—decoding, intervention, and ablation—are used to distinguish readout from computation, with strength-matched controls and a preregistered decision classifier.
Overall approach: separately test "the model's output about a role (readout)" and "the model's internal computation that assigns the role (computation)", running three sets of experiments on the workspace (J-space/R-space)—linear probe decoding, role-direction intervention, and workspace ablation—all with the stimulus family as the resampling unit.
Stimulus set and task
- Each stimulus family contains 4 minimal-pair sentences, crossing role (agent/patient) with the linear position of the target entity (first/second), covering 4 syntactic structures (active-passive, cleft, relative clause, dative).
- Within a family, averaging over positions cancels word-order effects, and averaging over structures removes voice/syntactic-frame effects, leaving role assignment as the remaining factor.
- The direction-fitting corpus and the evaluation corpus do not overlap at either the template level or the entity level.
- Readout method: append a role probe to the stimulus sentence (e.g., asking who performed the action), take the probabilities of the two tokens—the target entity and the other entity—in the full-vocabulary softmax, convert to log-odds, and base all subsequent measurements on these two probabilities.
Decoding test
- At the readout layer, take four activation sources from the residual stream: the full residual stream h, the J-space component (orthogonal matching pursuit on the J-lens dictionary, K=16), the orthogonal remainder (h minus the J-space component), and a same-rank random subspace (controlling for capacity differences).
- The probe is closed-form ridge regression onto ±1 role labels, with leave-one-pair-out cross-validation: an entire concept pair is held out, and exceeding 0.5 requires role information to generalize across lexical items.
- Two robustness checks: a quadratic probe and a random-Fourier-features probe (covering the product structure required for tensor-product binding); seven ridge penalty values are scanned under nested cross-validation.
Direction fitting and intervention
- For each entity and each position, fit the role direction r with two estimators: diff-of-means (mean of the agent class minus mean of the patient class) and the LRE gradient estimator, the latter taking the gradient of the readout with respect to the activations and pooling across the two roles: rLREentity = unit(1/n∑i ∇h (zentity − zother)), meaning the direction to which the readout actually responds rather than the direction of between-class separation.
- The intervention is h ← h + s·c·r, with s ∈ {+1, −1} selecting toward-agent or toward-patient, and c a calibration coefficient; the same (r, c) is applied to all cells within a family, not conditioned on the sentence's own role label.
- Three strength-matched controls: a non-participant direction (a fitted direction for an entity not appearing in the sentence), a random unit vector, and a shuffled-label direction; an unedited baseline is also collected.
- c is calibrated on the fitting corpus by (model, estimator, position), taking the smallest value on a doubling grid such that the toward-agent push moves the entity log-odds by at least 0.5 on patient-role sentences; directions must pass a bootstrap stability threshold (mean cosine ≥ 0.8, 200 resamples).
Binding score and statistics
- Binding score: BSi = −(dG(toward agent) + dG(toward patient))/2, where dG(s) is the change in the log-odds difference between the two roles after the push (minus the natural difference); under bag-of-concepts, dG(s) is near zero, whereas when binding exists the change in difference is negative for both signs and BS is positive.
- The same statistic from control edits is used to generate a null distribution, and a two-sided percentile band is taken; the pooled mean must fall outside the null band to be credible.
- The primary test is a two-sided sign-flip permutation test (agent/patient labels are exchangeable within a family), with Holm-Bonferroni correction; results are classified into four tiers by preregistered thresholds: positive binding, indistinguishable from controls, clean negative, uninterpretable strength failure.
Ablation test
- During the forward pass, zero out the projection of the residual stream onto the span of the most active J-lens vectors (K_ablate=10, selected per stimulus by matching pursuit); the control is a same-rank random-subspace ablation (redrawn per family).
- Measure the log-odds margin on two tasks: binding (role probe) and recall (concept probe, asking which participant matches an occupation cue, balanced between the two participants); the binding-specific deficit ∆ is the difference between the normalized drops of the two tasks' margins relative to their respective unedited baselines; for causality to hold, ∆ must exceed the 0.05 lower bound, exceed the random control, and have a bootstrap CI excluding zero, and the two tasks' baseline accuracy difference must be within 0.02.
What experiments did the paper conduct?
Three sets of experiments: J-space decoding is below a same-rank random subspace; LRE directions shift readout but do not move binding; ablation impairs binding only in Gemma-3-27B-IT.
Experimental setup
- Models tested: Qwen3.6-27B (64 layers, d_model=5120, base), Gemma-3-12B (48 layers, base), Gemma-3-27B-IT (62 layers, instruction-tuned, the only instruction-tuned model in the set).
- Lenses: Qwen and Gemma-3-12B use the Neuronpedia pre-fitted J-lens (WikiText-103, 1000 prompts); Gemma-3-27B-IT runs three lens variants (the same released J-lens, R-lens, and its accompanying J-lens).
- Stimulus set: the main set has 3 concept pairs (doctor/nurse, teacher/student, driver/passenger) × 4 structures × 2×2 role×position, with 50 families per cell, 600 families and 2400 sentences per model; there is also a Qwen six-pair robustness run (25 families per cell).
- Readout layer and position: the main readout layer is at about 75% relative depth (Qwen L48, Gemma-3-12B L36, Gemma-3-27B-IT L46), and intervention and ablation are only at that layer; probe/intervention positions are entity tokens (main results) or sentence-final tokens.
- Metrics and statistics: role decoding accuracy (chance=0.50), binding score, binding-specific deficit ∆, Cohen's d; bootstrap and permutation each with 10,000 resamples, α=0.05, Holm-Bonferroni correction; positive binding requires d ≥ 0.5, falling outside the null band, and consistency in at least 2/3 of agent/patient structures.
- Seeds: Qwen completed 3/5 seeds (0, 42, 7), and both Gemma models have all 5 seeds.
Main results
Table 1 (sentence-final position, within-pair role decoding accuracy, chance=0.50):
Swipe horizontally to read the table
Activation source Qwen3.6-27B Gemma-3-12B Gemma-3-27B-IT Full residual stream 0.989 0.975 0.978 Orthogonal remainder 0.984 0.949 0.974 Random subspace (same rank) 0.643 0.589 0.590 J-space 0.587 0.573 0.574 - The authors state that J-space accuracy is below the residual stream it is derived from and is at or below the same-rank random subspace (spread across models=0.014), i.e., role information lies outside J-space.
- The authors state that this null result is not a probe-capacity artifact: neither nonlinear probes nor the ridge scan made J-space exceed the random subspace, and this holds across seeds and across layers within the workspace band (Appendix B).
Intervention experiment (Section 5.2)
- What was measured: pushing the residual stream along fitted role directions, measuring readout shift and binding score. Under LRE, direction stability mean cosine is 0.967–0.999 (three models, two positions); diff-of-means at sentence-final position has stability 0.580–0.679, below the 0.8 threshold, and the comparison panel shows no readout shift.
- Results: in the Gemma-3-12B three-direction panel, readout shifts are agent-push +0.94, patient-push −0.99, random direction −0.01 log-odds. Table 2 (Gemma-3-27B-IT entity tokens, LRE coefficient scan): at c=64/128/256 the shifts are 0.328/0.645/1.255, pooled means −0.0134/−0.0568/−0.1809, non-participant controls −0.0161/−0.0561/−0.2161, ratios 0.83/1.01/0.84—the pooled effect grows more than tenfold but the control grows in step, always within the null band. Qwen3.6-27B (c=4) has pooled effect −0.0006 and control −0.0046, without the signed steering of the other two models; Gemma-3-12B (c=256) has pooled −0.0140 and control −0.0308.
- Authors' interpretation: the decision classifier returns indistinguishable from controls on all three models; the authors state that LRE directions move the role readout in a signed, reversible way, but their push on binding does not exceed the non-participant control.
Ablation experiment (Section 5.3, Table 3)
- What was measured: zeroing the top-10 J-space directions and measuring the binding-specific deficit ∆, with a same-rank random-subspace ablation as control.
- Results: at Gemma-3-27B-IT entity tokens, ∆=+0.275 (95% CI [0.262, 0.288], role accuracy 0.990→0.740, concept accuracy maintained at 0.920, residual-norm perturbation of the two edits differing by within 7%), and at sentence-final position ∆=−0.001; this asymmetry replicates across the three lens variants (binding deficits 0.284, 0.334, 0.297, Appendix D). Qwen3.6-27B shows no binding-specific damage at either position (entity +0.014, sentence-final +0.000), and its J-space edit perturbs the residual norm about 5 times as much as the random edit (ratios 5.39/4.97) yet still shows no extra binding damage; Gemma-3-12B shows +0.008 at sentence-final position and +0.273 at entity position, but concept accuracy falls together with role accuracy, failing the difficulty-matching precondition (marked †).
- Authors' interpretation: this is the only causal signal in the study; the authors state that it shows the ablation design can detect binding when binding exists, so the null results in the other two models are true null effects; however, this result is confounded between instruction tuning and model scale, and the authors do not resolve the attribution.
Other ablations and analyses
- Cross-pair transfer (Appendix C): at sentence-final position all activation sources invert to 0.28–0.32 (including the residual stream at 0.283), which the authors state reflects the probe setup rather than J-space; at entity tokens the residual stream is 0.594–0.644, the orthogonal remainder 0.603–0.626, and the random subspace 0.575–0.713, all above chance, while J-space is 0.510–0.524, at chance (Table 4).
- Nonlinear probes (Appendix B): with the quadratic probe on J-space, Qwen entity tokens are 0.522 and Gemma-3-12B 0.449; under every probe family the random subspace is above J-space.
- Ridge scan (Appendix B): J-space rises to at most 0.645, while the random subspace under the same scan rises to 0.678–0.724.
- Seeds and layers (Appendix B): Gemma-3-12B random subspace across 5 seeds is 0.313–0.330; at L18, J-space and the random subspace are on par (0.323 vs 0.322).
- Identity-swap strength check: counterpart probability shifts across the three models are 0.0012 (Qwen), 0.0048 (Gemma-3-12B), 0.0001 (Gemma-3-27B-IT), all below the 0.05 lower bound; calibration fails, so this run uses full swaps rather than the calibrated scaled version.
What are some points that can be further explored?
The authors list limitations including the failed strength check and the confounding between instruction tuning and scale; the experiments cover three models, English agent-patient stimuli, and a single layer per model.
Limitations and future directions noted by the authors
- The identity-swap intervention strength check failed on all three models, limiting how far the null steering result can be pushed (Limitations); the authors propose a follow-up positive control: patch the residual of the opposite-role sentence at the same position and calibrate the push accordingly.
- The positive ablation in Gemma-3-27B-IT is confounded between instruction tuning and model scale (Limitations); the authors propose that running the ablation on the base version of Gemma-3-27B would resolve this.
- All three models are dense transformers, the stimuli are English agent-patient, and there is a single layer per model; whether this dissociation holds in other architectures, layers, languages, or richer relational structures (coreference, temporal relations, multi-entity planning) was not tested (Limitations).
- The authors propose comparing with Natural Language Autoencoders [Fraser-Taliente et al., 2026] on Gemma-3-27B-IT, which could test whether this dissociation is specific to the workspace-lens paradigm or general to verbalizable readouts (Limitations).
- The authors state that whether the same holds in real monitoring scenarios remains open (Conclusion).
Scope of experiments
- The models tested are Qwen3.6-27B, Gemma-3-12B, and Gemma-3-27B-IT, 3 in total, of which 1 is an instruction-tuned model (Section 4).
- The stimuli are English agent-patient role bindings, 3 concept pairs (Qwen additionally ran 6 pairs) × 4 syntactic structures, 600 families and 2400 sentences per model (Section 4).
- Interventions and ablations are applied only at a single readout layer per model (about 75% depth); the layer sweep is done only on Gemma-3-27B-IT (Section 4).
- The R-lens experiments are run only on Gemma-3-27B-IT (three lens variants); the other two models use only J-lens (Section 4).
- Number of seeds: Qwen3.6-27B completed 3 of 5 seeds, and the two Gemma models have all 5 (Appendix E).
Summarize the main content of the paper
The authors state that the workspace carries a role readout direction rather than a binding substrate, and that a monitor based on it would miss the role assignments represented by the model itself.
An interpretability analysis paper that tests whether the verbalizable workspace proposed by Gurnee et al. [2026] carries role-filler binding—the structure a runtime alignment monitor built on it would need to distinguish propositions sharing a concept set (such as "the model plans to deceive the user" and "the user plans to deceive the model").
- Design: on Qwen3.6-27B, Gemma-3-12B, and Gemma-3-27B-IT, using two lenses (J-lens/R-lens) and two direction estimators (diff-of-means/LRE), runs three sets of experiments: decoding, intervention, and ablation, with strength-matched controls such as same-rank random subspaces and non-participant directions, and a preregistered decision classifier.
- Decoding: residual-stream role decoding is near ceiling (end-of-sentence 0.975–0.989), while J-space (0.573–0.587) is at or below same-rank random subspaces (0.589–0.643, Table 1); nonlinear probes and regularization sweeps do not close the gap.
- Intervention: LRE directions move role readout in a signed way (+0.94/−0.99 log-odds on Gemma-3-12B), but the binding score is indistinguishable from non-participant controls, and all three models are judged indistinguishable from controls.
- Ablation: binding-specific impairment of +0.275 (CI [0.262, 0.288]) appears only at entity tokens in Gemma-3-27B-IT, replicated across three lens variants; the authors state that this shows the design can detect binding and that the nulls in the other two models are true null effects, but this positive is confounded between instruction tuning and scale.
- Authors' conclusion: the workspace supports role readout but does not carry the computation of binding; a monitor based on workspace directions surfaces the model's role report rather than its binding, and will respond similarly to both on propositions that share a concept set in benign and problematic states.