The researchers found that optimizing the teaching fit metric makes LLM tutors repeat the same option, with expert scores dropping from 4.46 to 3.03

Optimizing with a pedagogical adaptivity metric can raise the metric while lowering expert-assessed teaching quality.

• Scenario: The authors argue that when large language models take on tutoring roles, the personalization promise depends directly on pedagogical adaptivity, i.e., adjusting teaching decisions to the given learner and context. Content errors are easy to see, while teaching misalignment requires systematic analysis; the authors call "stating the correct principle but selecting an activity that contradicts it" pedagogical drift.
• Limitations of existing evaluations: The authors claim existing evaluations decompose tutoring into independent capabilities like prompting, correction, and scaffolding, so a model can do each step correctly yet never adapt to the context; evaluations also look only at external behavior, while chain of thought often does not reflect the computation driving outputs.
• Strategy tested: After detecting misalignment, the authors say the natural response is to define a metric for the missing quality and fine-tune on it. The authors argue whether this improves teaching depends on whether the metric captures quality in expert judgment; automated scoring and human evaluation of educational outputs often diverge.
• Core observation and method: The authors design the metric following the rubric format used in the field: each teaching choice is scored against conditions held constant in the case, then averaged. The authors use this metric to audit a frontier tutoring model, correct the weakest cases on an open-source agent, have trained educators blindly rate before and after correction, and use weight analysis to trace where the correction lands.

Theoretical basis for instructional adaptation

• Zone of proximal development and scaffolding: Vygotsky (1978) and Wood, Bruner, and Ross (1976) place effective teaching at the boundary between what a learner can do alone and what they can do with support; van de Pol et al. (2010) specify that scaffolding includes contingency, fading, and transfer of responsibility. The authors use these theories to support content complexity and support structures.
• Schemas and depth of engagement: Piaget (1952), Posner et al. (1982), and Chi (2008) distinguish construction, assimilation, and accommodation; Chi and Wylie's (2014) ICAP ranks engagement depth, and the authors claim this ranking holds only for pattern-matching tasks and learners.
• Changing learners: Corbett and Anderson (1994) model knowledge state as evolving with each instructional action. The authors emphasize that knowledge state, learning phase, and learning context are positions on a trajectory, not fixed labels.

Evaluation and optimization of tutoring models

• Move-level benchmarks: Macina et al. (2025), Shi et al. (2026), and Maurya et al. (2025) evaluate teaching quality at the tutoring move level. The authors claim these evaluate prompts, corrections, scaffolding, etc., independently of learning conditions.
• LearnLM rubric: LearnLM Team et al. (2025) is cited by the authors as a line of learning-oriented models in educational products; the authors claim such rubrics weigh the tutor side more, rather than whether decisions align with the learner's context.
• Optimizing tutoring with rewards: Sonkar et al. (2024) and Dinucu-Jianu et al. (2025) optimize tutoring via simulated students, preference optimization, or reinforcement learning, with rewards judged by LLMs as good scaffolding moves. The authors claim they examine the effect of such optimization when scores are replaced with adaptivity metrics.

Human-machine judgment and construct validity

• Inconsistent comparison standards: Pal Chowdhury et al. (2025) and LearnLM Team et al. (2025) are used by the authors to illustrate that existing human-machine comparisons often do not compare both sides on the same standard.
• Disagreement in specialized knowledge: Szymanski et al. (2025) report that automated scoring and expert judgment diverge where specialized knowledge is needed. The authors claim existing literature documents disagreement but does not characterize where it appears across instructional dimensions and conditions.
• Construct validity checks: Bean et al. (2026) review 445 benchmarks, and the authors claim construct validity is rarely established, proposing a checklist. The authors later argue this checklist certifies measurement, not an optimizable objective.

Interpretability of chain-of-thought and fine-tuning

• Chain of thought: Barez et al. (2025), Chen et al. (2025), and Lanham et al. (2023) discuss that chain of thought often does not reflect the computation driving outputs. The authors conclude that querying the model's reasons alone cannot determine whether it is doing pedagogical reasoning.

• Localizing fine-tuning: Burns et al. (2024), Cywinski et al. (2025), Xue and Mirzasoleiman (2026), and Shuttleworth et al. (2025) are cited to support that interpretability methods can see where and how corrective fine-tuning changes behavior.

• Authors' prior audit: Domínguez et al. (2026) describe pedagogical drift. In the discussion, the authors claim that this audit's model difficulties at Bloom's analyze and evaluate levels reproduce the patterns of that audit with another set of metrics, but the agent did not exhibit this.

• Baselines and corpus: The control arms are the audited model's original outputs, an unfinetuned Qwen3-32B, a diversity-preserving adapter that changes only one decision point, and 8 high-PAI control cases; decision-point scores are also compared against the expected score of uniform random options. The corpus is the authors' own 2,000 factorial learning contexts, not the above move-level benchmarks.

The authors position this paper in the gap that "conditional adaptation has not been examined": existing work either evaluates isolated moves or uses LLM judges to reward scaffolding. In the discussion, the authors further contrast with Goodhart (1975), Strathern (1997), Gao et al. (2023), and Skalse et al. (2022), claiming the degradation here comes from the form of separable objectives averaged over fixed contexts, not from the optimizer finding unexpected shortcuts.

The overall approach is to first build a Pedagogical Adaptivity Index (PAI) that is isomorphic to the domain's automatic scoring, then use it simultaneously as an audit tool and a correction target, and finally use educator blind evaluation and weight decomposition to separate "score increase" from "whether teaching improved."

Elicitation and scoring

• Input: Each scenario is an 80–120 word case narrative that implicitly includes knowledge state, learning stage, learning context, the cognitive level of the objective (three Bloom levels), and the subject family. The task is the same for all cases: design five learning activities, each selecting one option at each of five decision points and providing a brief explanation. Output is constrained JSON, and choices are directly parseable.
• Five decision points: content complexity, cognitive task, support structure, engagement mode, and subject approach. Each point has four ordered options, each governed by conditions identified by the authors in the learning sciences, totaling eleven alignment submatrices (Table 1, Appendix A). The matrices are frozen before any output is scored.
• Aggregation: The score for a single decision is the mean of the applicable two or three matrix lookups; the activity score is the mean of the five decision scores for that activity; the PAI for a response is the mean of the five activity scores, hence also the mean of the 25 choice scores (Table 2). The scale is −1.0 to +1.0. The index scores only choices; explanations are shown to educators but do not enter any score.
• Structural properties: Each decision is scored independently, and case conditions are constant across the five activities, so there are no cross-activity variation, progression, or coherence terms. The authors also compute repetition rate (proportion of choices equal to the modal option) and choice entropy (Shannon entropy of the distribution), averaged over the five decision points; these do not enter the correction signal. When all choices are identical, repetition rate is 1.0 and entropy is 0.

Cases, models, and two corrections

• Models: The audit model is Gemini 3.1 Pro Preview; the authors state they chose it because the Gemini family includes LearnLM for educational products. All fine-tuning is done on Qwen3-32B. The authors state that the mechanistic conclusions pertain to PAI correction on open-source instruction-tuned transformers, not any commercial tutoring model. Case narratives are generated by grok-3-fast.
• Four sets: A five-condition full factorial sample yields 2,000 balanced scenarios, with checks for faithfulness, absence of answer hints, and deduplication (Appendix B). The intervention set is 30 cases with PAI≤0.07; the control set is the 8 cases with highest PAI; the correction training set has 589 cases, stratified by the ten weakest condition cells of the audit model, including the intervention set and excluding the control set; the out-of-sample set has 200 cases, disjoint from the correction set.
• Greedy signal: At each decision point in each activity, select the option with the highest matrix score, ignoring other criteria. Explanations come from 100 fixed phrases unrelated to the learner, with five paraphrases per option, so the five activities have different wording but identical choices.
• Diversity-preserving signal: For each case, only one decision point is changed toward the metric; the other decisions retain the originally produced choices, and cross-activity variation is preserved. One adapter is trained per changed decision point, totaling four.
• LoRA: Applied to all linear modules; each adapter has 449 adapted matrices. Greedy ranks r are 1, 4, 8, 16, with α fixed at 32, so the effective scale α/r varies with rank. Learning rate is 1×10⁻⁴, constant, 3 epochs, batch size 1, Adam, dropout 0, seed not fixed. Loss is cross-entropy on the final assistant message. Human evaluation uses r=8 greedy outputs. Audit decoding temperature is 0.7; proxy queries on the 589 cases use temperature 0.7; evaluated cases and out-of-sample cases use greedy decoding.

Expert evaluation

• Sample: 51 people were invited; 31 completed both rounds. All have education degrees and are enrolled in an AI and education graduate program. Participation is voluntary and ungraded. Each person evaluates 8 cases; the design assigns 10–13 raters per case, with actual independent ratings per case ranging from 4 to 11, mean 6.5.
• Items: Six items on a 1–5 scale covering overall alignment and the five decision points, plus two open-ended questions on weakest and strongest aspects (Appendix C gives full wording). Raters are blind to the metric, which cases were modified, and the existence of manipulation. Control cases are unchanged across rounds; case order is re-randomized.
• Text coding: 992 open responses are rule-matched against eleven code categories. Two educators independently code a stratified 20% (n=99) of the weakest-aspect texts, blind to the automatic coding. The authors pre-specify that only codes with at least moderate agreement both between coders and between coders and the program are used.

Weight analysis and condition recovery

• Experiment 5: For each adapted matrix, compute the update ΔW=BA, take the Frobenius norm, normalize element-wise, and partition by network depth and module type (attention q/k/v/o, MLP gate/up/down, output projection), and examine spectral profile and effective rank. The authors pre-specify: if updates concentrate at the output level, are achievable at low rank, and are roughly uniform across depth, this is interpreted as shallow behavioral editing rather than deeper representational change.
• Experiment 6: For each adapted matrix, compute the principal angles between the greedy and diversity-preserving update subspaces on the input and output sides, and the proportion of greedy update energy lying in the span of the latter. The authors pre-specify: if overlap is large, the metric gain and accompanying degradation are bound in the same direction; if directions are separated, in principle one could target only one direction.
• Post-hoc probe: The 200 out-of-sample narratives are fed back to the model, classified by the five conditions using fixed order, constrained JSON, greedy decoding; both audit model and proxy responses are parsed. Additionally, TF-IDF plus logistic regression with five-fold cross-validation estimates the recoverability of each condition from the narrative text itself, reporting accuracy, majority-class baseline, and Cohen's κ.

Experimental setup

• Models: Audit Gemini 3.1 Pro Preview; fine-tuning, out-of-sample behavior, and weight analysis all on Qwen3-32B; narratives generated by grok-3-fast (temperature 0.8, max 300 tokens). No LLM judges.
• Corpus: 2,000 factorial contexts. Expert materials are 38 cases: 30 intervention examples (PAI≤0.07) and 8 highest-PAI controls. Correction training on 589 examples, out-of-sample 200 examples.
• Arms: Audited model's original outputs; greedy LoRA (r=1, 4, 8, 16); four diversity-preserving adapters that change only one decision point (r=8); unfinetuned base proxy. In the second expert round, greedy r=8 replaces intervention examples; control examples are unchanged.
• Metrics: PAI (−1 to +1); repetition rate and choice entropy not entering the training signal; expert 1–5 scores on six items; Krippendorff's α; Cohen's κ for open text. Decision points also report differences above the expected score of uniform random options.
• Human ratings: 31 people complete two rounds, 8 cases each, 4–11 ratings per case, mean 6.5. Blind to metrics, modifications, and manipulation. Statistics are Spearman, Mann–Whitney U, and Wilcoxon.
• Training and decoding: 3 epochs, learning rate 1×10⁻⁴, batch size 1, seed not fixed. Audit temperature 0.7; agent arms on 589 examples temperature 0.7; rated cases and 200 out-of-sample cases temperature 0.0.

Main results

The table below shows the 30 intervention examples and 8 control examples used in expert comparisons. Section 4.2 writes the audited PAI of intervention examples as +0.05; Section 4.3 writes 0.051 for the same 30 examples and gives base proxy 0.059. Second-round control outputs are unchanged; their PAI is not reported in the paper.

Output	Average PAI	Expert total score (1–5)
Gemini original output, 30 intervention examples	+0.05	4.46
Qwen3-32B greedy r=8, same 30 examples	+0.42	3.03
Gemini original output, 8 control examples, first round	+0.59	4.39
Same control examples, second round (outputs unchanged)	Not reported	4.51
Qwen3-32B base proxy, same 30 examples (objective is the proxy)	0.059	Not reported

• Opposite directions: The authors claim these 30 examples are the lowest-scoring in the corpus with the most room for improvement, so human ratings are conducted under the most favorable conditions for intervention. Wilcoxon p < .00001 for total score decline, and p < .00001 for each of the six items individually. Controls go from 4.39 to 4.51 (p=.38), which the authors claim rules out rater drift, fatigue, or overall changes in rating behavior. Spearman ρ=−0.24 (p=.21) between metric gains and score declines on cases; the authors claim the two are decoupled, not simply inversely related.
• Raters note repetition: α rises from about 0 to 0.27, and the proportion of ratings 4 or 5 falls from 90% to 52%. Two coders have κ of 0.65 and 0.63 only on tutor role and monotony; other codes have κ≤0.47. Monotony appears in 19–25% of corrected intervention examples' weakest-aspect texts, at most 3% in original outputs, and in one control text; automated program gives 16%, others 0 (κ=0.48–0.59 with coders). Corrected outputs flagged for monotony have total score 2.2, unflagged 3.2; the paper does not give text counts for these two groups.
• Another difference: Corrected explanations come from 100 learner-independent phrases, word count 11.9 vs. 16.7 for original outputs, and the proportion of words shared with the narrative drops from 0.21 to 0.03. The authors claim the score decline cannot be fully decomposed into "option collapse" and "generic explanations." Base proxy on the same examples has word overlap 0.23 and no monotonic sequences; educators did not rate base proxy.

Baseline audit

On 2,000 examples, the audited model's average PAI is +0.29 (SD 0.11); the authors claim 96% fall between 0 and +0.5, with very few negative cases (Figure 3a). The authors claim that when decomposed by condition, the lowest is novice in the skill-building phase, and the intervention set consists mainly of that combination (Figure 3c).

Swipe horizontally to read the table

Decision point	Gemini above random	Qwen above random	Random baseline
DP1 Content complexity	+0.143	−0.030	+0.146
DP2 Cognitive task	+0.060	−0.045	+0.135
DP3 Support structure	+0.036	−0.060	+0.209
DP4 Engagement mode	+0.197	+0.079	+0.167
DP5 Disciplinary approach	+0.128	+0.079	+0.246

From Table 3. Gemini on 2,000 examples, Qwen on 589 examples. The authors claim only the above-random columns are comparable across decision points, and that the Spearman ρ=0.900 between the two columns' rankings indicates the profile is not specific to one model family. In the table, Qwen's DP1–DP3 above random are negative, Gemini's positive. Pre-correction, case-average expert scores and PAI have ρ=−0.15 (p=.39), with all five decision points |ρ|≤0.23, p≥.17; the lowest 30 and highest 8 examples have expert scores 4.46 and 4.39 (p=.53).

Degenerate optimum and out-of-sample

• Objective itself fully monotonic: Because each activity is scored independently against fixed conditions, each decision point has a highest-scoring option, and the highest-scoring sequence across five activities is that option repeated five times. The correction objective has repetition rate 1.000 and choice entropy 0.000. The authors claim this comes from the metric structure, not from how the objective text is written.
• All four ranks collapse: Greedy models are 100% fully monotonic on tested cases at r=1, 4, 8, 16. Appendix B writes this result at temperature 0.7 on 589 correction examples, and writes the out-of-sample 100% under greedy decoding. r=1 means each adapted matrix has only one direction; the authors claim no high-dimensional change is needed.
• Not a model swap: On 589 examples, base proxy and audited model have average PAI 0.184 vs. 0.197, repetition rate 0.490 vs. 0.486, entropy 1.540 vs. 1.536, and base proxy has no monotonic outputs. On 30 intervention examples, 0.059 vs. 0.051, repetition rate 0.481 vs. 0.483. On 200 unseen examples, greedy corrected model is again 100% fully monotonic, average PAI 0.42, base proxy 0.30; the paper does not state the rank in this comparison. On DP2, all 200 unseen learners receive the same option DP2-b, decision-point score drops from 0.24 to 0.04; the authors claim average PAI still rises because gains on the other four decision points mask this loss (Figure 6).

Weight localization and condition recovery

• Readout layer: The proportion of update energy in the output projection is 75% at r=1, 92% at r=4, 89% at r=8, 93% at r=16. After element-wise normalization, this projection's update is 18–30 times that of any transformer block; energy across 64 blocks by depth quartile is 23–31%, with uniform baseline 25% (Figure 7). Of 750 explanations on intervention examples, 748 are verbatim from 100 training phrases. The authors claim the correction rewrites the readout, not the computation producing outputs.
• Direction of two supervisions: Diversity-preserving output projection share is 66–80%, engagement-mode arm 41%, with more MLP involvement. Greedy update energy has only 4.5% in the subspace spanned by diversity-preserving; output-side directions share 48.8%, input-side share 3.3%. The authors claim both share the part that pushes up the same option token on the output side, and share almost nothing on the input side.
• Which conditions are recoverable: On 200 out-of-sample narratives, κ for discipline family, learning context, and knowledge state is 1.000, 0.970, 0.977 for the audited model, and 1.000, 0.985, 0.930 for the agent; cognitive level is 0.731 and 0.663; learning phase is 0.157 and 0.046, with the agent's 95% interval [−0.03, +0.11] containing 0 (Table 4). A bag-of-words classifier has κ 0.149 for learning phase and 0.908 for cognitive level. The authors' average recoverability order for governing conditions is DP3 0.97, DP1 0.83, DP4 0.59, DP5 0.55, DP2 0.40, with ρ=+0.95 (p=.014) against the corrected model's number of distinct options used on 200 learners (3, 3, 2, 2, 1); with baseline adaptation, ρ=−0.10 (p=.87). The authors claim recoverability predicts where metric-driven collapse lands, not pre-correction capability.

Other ablations and analyses

• Diversity-preserving PAI on 589 examples: content +0.20, task +0.24, support structure +0.25, engagement mode +0.10; the same paragraph's text writes base proxy 0.185 and greedy 0.40, without stating greedy's rank (Figure 4a).
• Pre-specified condition evolution rescoring: correction's advantage over original outputs shrinks from +0.37 to +0.07 (0.34 vs. 0.27, p=.047); faster and slower schedules from −0.02 (p=.46) to +0.15; ρ=−0.09 with expert score changes. Only 45 of 432 condition cells still have fully monotonic optima.
• Final training loss: r=1/4/8/16 is 0.0028/0.0031/0.0029/0.0030 (Appendix B). The authors claim this is the same result as memorizing 589 monotonic objectives, described two ways.
• Learning phase misclassification: 38 of 64 competency development cases and 28 of 45 comprehensive mastery cases are labeled conceptual orientation.
• The authors list in Table 5 within-tier recovery rates from 0.59 to 0.97, and claim the model treats Analyze, Evaluate, and Create differently; the main text does not give a per-tier breakdown table.
• Baseline free text: The authors claim under human coding about nine-tenths of weakest-aspect criticisms fall within the five decision points, but due to lower consistency on other codes, only tutor role and monotony are used later.

Limitations and future directions noted by the authors

• Human evaluation did not cover the base proxy: Weights are closed-source; representation analysis uses an open-source proxy. The authors state that the base proxy is indistinguishable from the audit model on the metric and sequence diversity, but no educator has evaluated it, so the causal claims from human evaluation rely on computational equivalence, not human ratings (Section 5.4).
• Score decline has two simultaneous changes: The corrected outputs seen by educators have both collapsed selections and generic explanations from the fixed phrase library; the design cannot fully attribute the decline to either. The authors propose as a follow-up to reproduce with corrected selections paired with the model's own explanations (Section 5.4).
• Human evaluation is within the training set: The 30 intervention cases belong to the correction training set; educators evaluated cases the model was trained on. The authors attribute out-of-sample generalization only to behavioral collapse, not to human evaluation (Section 5.4).
• Coding, raters, and synthetic narratives: Qualitative coding is rule-based, only two coders, and substantial agreement only on two codes; raters are graduate students, not practicing teachers; narratives are synthetic, with one condition weakly coded. The authors state that synthetic stimuli should be human-validated before scoring (Section 5.4).
• Real trajectories and separability: The authors state that teaching quality is realized in real learners whose knowledge states change; abstracting this away could certify capabilities that do not withstand real teaching. Weight separability is only descriptive; it has not been proven that removing orthogonal components preserves gains and removes collapse; the training form of the diversity-preserving objective is not released, only the outputs and weights of the four adapters (Section 5.4).
• Seeds, scale, and binning: Training seeds are not fixed, each condition is trained once, and fixed α makes rank and update scale covary; the authors state that main findings do not depend on differences between bins because each bin fully collapses and is reproduced on 200 held-out cases. The index collapses Bloom's six levels into three bins; the authors state that results depend on this binning. The extensions the authors write include: reproducing on a second open-source model to determine whether collapse is a property of the objective; using real curricula and practicing teachers, and including sequence terms in the tool from the start; projecting greedy updates inside and outside the shared subspace and then testing behavior (Section 5.4). In the discussion, the authors also state that the design does not answer whether adaptation in the un-fine-tuned model is based on representations of learner conditions or shallower associations (Section 5.2).

Experimental coverage

• Models and generation: Audit on 2,000 cases with Gemini 3.1 Pro Preview; fine-tuning, out-of-sample behavior, and weight analysis on Qwen3-32B; narrative generation model is grok-3-fast (Section 3.3, Appendix B).
• Human evaluation scale: 38 cases, two rounds, 31 graduate students with education degrees completed, 4–11 ratings per case, mean 6.5; the paper states that no educator evaluated the base proxy (Sections 3.5, 5.4).
• Correction budget: Training set 589 cases, out-of-sample 200 cases; greedy ranks 1, 4, 8, 16, plus four adapters each modifying only one decision point; each condition trained once, seeds not fixed (Sections 3.3–3.4, Appendix B.7).
• Agreement numbers: The paper reports Krippendorff's α for both rounds, and κ for the stratified 20% (n=99) of weakest-aspect texts by two educators; the number of texts with monotonic scores 2.2 and 3.2 in the two groups is not reported (Sections 3.8, 4.2).
• Mechanism and probes: Weight analysis covers LoRA updates in attention, MLP, and output projection; condition recovery uses greedy classification on 200 out-of-sample narratives for both models, and trains a bag-of-words classifier (Sections 3.6–3.7, Table 4).

After making the pedagogical adaptivity metric a fine-tuning objective, the metric rises while scores from trained educators fall.

• Problem: The authors aim to answer two things: whether optimizing the pedagogical adaptivity metric improves the teaching it measures; and whether the correction occurs at the model's output level or deeper in its computation.
• Tools: PAI uses eleven frozen matrices, scores choices on five decision points independently and averages them, with case conditions unchanged across five activities. Repetition rate and choice entropy are computed but do not enter the correction signal.
• Method: Audit Gemini 3.1 Pro Preview on 2,000 factorial contexts, then perform LoRA on Qwen3-32B with an objective maximizing this metric. 31 educators blindly rate 30 weakest examples and 8 control examples before and after correction.
• Human ratings: greedy r=8 moves the average PAI on these 30 examples from +0.05 to +0.42, expert total score from 4.46 to 3.03, with Wilcoxon p < .00001 on all six items. Unchanged controls go from 4.39 to 4.51 (p=.38). Pre-correction expert scores do not correlate with PAI (ρ=−0.15, p=.39).
• Sequence: The authors claim the metric's optimum is to repeat the highest-scoring option for each decision point five times. Greedy correction on tested cases across four ranks, and on 200 unseen examples, is 100% fully monotonic. On unseen examples, DP2 score drops from 0.24 to 0.04, average PAI remains 0.42, and base proxy is 0.30. When conditions evolve along the sequence, the advantage shrinks from +0.37 to +0.07; the authors claim no reversal and no closer alignment with expert score changes (ρ=−0.09).
• Authors' conclusions: At r=8, 89% of update energy is in the output projection, and 748 of 750 explanations are verbatim from a training phrase bank. The authors claim measurement validity does not imply optimization validity because optimization enters argmax that validation does not access; and in Table 5 they list design principles, including verifying whether conditions can be recovered from stimuli, scoring sequence properties separately, measuring adaptivity as decision differences, and publishing what full-score sequences look like.