Where does alignment emerge, and what does it mean? A closer look at the Platonic Representation Hypothesis

Core question: at which layers and which dimensions of models does representational alignment between unimodal vision and language models (the Platonic Representation Hypothesis, PRH) occur, whether its relationship with model capability still holds for new models, and how the absolute magnitude of cross-modal alignment scores should be interpreted.

• Background: PRH (Huh et al., 2024) holds that sufficiently strong vision and language models learn similar abstract representations, an idea that has inspired a set of methods for aligning pretrained unimodal models with lightweight alignment layers and a small amount of paired data.
• Shortcomings of existing work (as stated in the paper): Koepke et al. (2026) state that some conclusions depend on the evaluation protocol, and that the performance-alignment trend on early language models no longer extends to new model families; Gröger et al. (2026) argue that similarity scores are hard to interpret without suitable baselines; existing studies usually report only the maximum similarity over all layer pairs, obscuring at which layers and dimensions alignment occurs.
• The paper's approach: analyze along three axes—locate alignment layer-wise and dimension-wise (PCA compression); examine the effect of caption characteristics on the capability-alignment relationship; and contextualize cross-modal alignment using randomly initialized models, explicitly aligned VLMs, and same-modality model pairs as baselines.
• The authors' positioning: the authors state that, to their knowledge, no prior work simultaneously provides comparisons with randomly initialized models, explicitly aligned VLMs, and same-modality model pairs.

Proposal and Revision of PRH

• Huh et al. (2024)—proposed the Platonic Representation Hypothesis, holding that the representation spaces of neural networks are converging, and reported a positive correlation between language model capability and cross-modal alignment on WIT.
• Koepke et al. (2026)—pointed out that some PRH conclusions depend on the evaluation protocol, and that the performance-alignment trend on early language models does not extend to new model families; also found that with fixed k, alignment decreases as the gallery grows, but is stable when the k/N ratio is maintained. This paper follows their evaluation setup and uses their findings as a comparison.
• Gröger et al. (2026)—argue that similarity scores need to be calibrated against a "no alignment" null hypothesis, and propose calibrated mutual k-NN; this paper uses this calibrated variant.

Cross-modal Alignment Methods Based on Pretrained Unimodal Models

• Huang et al. (2026), Maniparambil et al. (2025), Roschmann et al. (2026), Vouitsis et al. (2024), Yacobi et al. (2025), Zhang et al. (2025a)—align pretrained unimodal models with lightweight alignment layers and limited paired data, rather than training cross-modal alignment from scratch (e.g., the CLIP and SigLIP series). Among them, Zhang et al. (2025a) report that their mapped representations outperform CLIP in a shared embedding space; this paper cites this result in Section 3.3 as supporting evidence.

Comparison Baselines and Benchmarks

• Baseline models: explicitly aligned VLMs—CLIP ViT-L/14, SigLIP 2 So400M, Qwen3-VL-Embedding-8B; additionally, randomly initialized models and same-modality model pairs (DINOv2–DINOv3, NV-Embed-v2–Qwen3-Embedding-8B) serve as references.
• Benchmarks/datasets: WIT, CC3M (DreamLIP captions), COCO, DOCCI; language capability is measured with MMLU, ARC Challenge, and GSM8K (evaluated via the LM Evaluation Harness).

The authors position this paper as a refined test of PRH: building on the critiques of Koepke et al. and Gröger et al., it adds layer-wise/dimension-wise localization, caption characteristic controls, and more complete baseline comparisons.

Overall approach: on four image-text datasets with different caption characteristics, use mutual k-NN overlap and its calibrated variant to measure the representational alignment between unimodal vision and language models, analyzing from three angles: "where alignment occurs," "its relationship to capability," and "how to interpret the scores."

Models and Data

• Vision models: DINOv2 ViT-L, DINOv3 ViT-L, MAE ViT-L; language models: 21 generative models from the BLOOMZ, Llama, Mistral, Qwen, Gemma, and DeepSeek families, plus the text embedding models Qwen3-Embedding-8B and NV-Embed-v2.
• VLM references: CLIP ViT-L/14, SigLIP 2 So400M, Qwen3-VL-Embedding-8B, taking representations in the joint embedding space after the final projection layer; the text encoder context limits of CLIP and SigLIP 2 are 77 and 64 tokens respectively, and overlong captions are embedded in chunks and then averaged.
• Datasets: WIT (short and noisy Wikipedia captions), CC3M (short synthetic DreamLIP captions), COCO (concise human captions), DOCCI (long, fine-grained human captions). Additionally, Qwen3-8B is used to summarize DOCCI captions to approximately 10%, 25%, 50%, and 75% of their original length, to control caption detail under the same set of images; the summarization prompt and examples are in Appendix H.

Measurement Methods

• Metric: mutual k-NN overlap (Huh et al., 2024) and the calibrated variant of Gröger et al. (2026). For each dataset, N = 1024 image-text pairs are sampled, comparing the index overlap between the k = 10 nearest images in vision space and the nearest captions in language space.
• Representation extraction: vision models use the CLS token, language models use mean-pooling, and all are L2-normalized.
• Robustness: Appendix G ablates k ∈ {1, 5, 10, 100} and (k, N) combinations with fixed k/N ≈ 0.01 on DOCCI; the authors state that the relative ordering across regimes is unchanged.

Layer-wise and Dimension-wise Analysis

• Layer-wise heatmaps: compute mutual k-NN between all layer pairs of each model pair, rather than reporting only the maximum (Figure 1, Appendix B).
• PCA compression: fit PCA independently for each layer of each model, transform language and vision representations separately, then compute mutual k-NN; examine the effect of the retained variance proportion and the number of retained dimensions on alignment (Figure 2, Appendix C).

Capability-Alignment Relationship

• Following the setup of Koepke et al. (2026), compute Pearson correlations between five-shot MMLU accuracy, four-shot ARC Challenge accuracy, five-shot GSM8K exact-match (all measured with the LM Evaluation Harness) and the alignment scores with DINOv3.
• For each dataset, 1024 image-text pairs are sampled independently and repeated 5 times, reporting mean and standard deviation; the WIT subset provided by Huh et al. has only 1024 samples, corresponding to a single run.

Baseline Contextualization

• Compare four regimes: randomly initialized models, unimodal vision-language model pairs (the main subject of study), explicitly aligned VLMs (cross-modal references), and same-modality model pairs.
• Appendix E (Table 1) verifies that the difference between using projection-layer representations and using the maximum layer-pair representations for VLMs is at most 0.016 (SigLIP 2, COCO), and does not change the regime ordering.

Experimental Setup

• Models evaluated: on the vision side, DINOv2 ViT-L, DINOv3 ViT-L, MAE ViT-L; on the language side, 21 generative models (BLOOMZ, Llama, Mistral, Qwen, Gemma, DeepSeek families) plus Qwen3-Embedding-8B and NV-Embed-v2; references are CLIP ViT-L/14, SigLIP 2 So400M, Qwen3-VL-Embedding-8B, and randomly initialized models.
• Datasets: WIT, CC3M (DreamLIP captions), COCO, DOCCI, and 10%/25%/50%/75% summarized versions of DOCCI.
• Metrics: mutual k-NN (k = 10, N = 1024) and calibrated mutual k-NN; the capability-alignment relationship uses Pearson r and p-values.
• Number of repetitions: in Section 3.2, 1024 pairs are sampled independently per dataset, with 5 runs reporting mean and standard deviation; the WIT subset has only 1024 samples and is a single run; Figure 5 and Table 1 use 3 runs (single run for WIT).
• Capability evaluation: MMLU (five-shot), ARC Challenge (four-shot), GSM8K (five-shot exact match), using the LM Evaluation Harness.

Main Results

Layer-wise alignment in Section 3.1 (Figure 1, WIT, mutual k-NN):

Swipe horizontally to read the table

Model pair	Peak	Peak location	Source
DINOv3 × Qwen3-8B (cross-modal)	0.143	DINOv3 layer 21 × Qwen3-8B layer 30	Figure 1a
DINOv2/v3 × MAE etc. (vision-vision)	0.538	Later layers	Figure 1b
Qwen3-8B × NV-Embed-v2 (language-language)	approx. 0.5	Multiple early-to-middle layer pairs	Figure 1c

• The authors state that cross-modal alignment occurs mainly at later layers, and that final-layer representations never reach maximum alignment, and may even be far from the maximum.
• Vision-vision forms three block-diagonal regions above 0.3; language-language (generative × embedding models) reaches approximately 0.5 at early-to-middle layer pairs.

PCA Compression Analysis

• What was measured: alignment of DINOv2 with 6 language models on WIT after PCA compression (Figure 2).
• Results: when 40% of variance is retained, alignment for most models is maintained or slightly improved; compression to approximately 100 dimensions still maintains cross-modal alignment (the original language space is 1,536–5,120 dimensions); stronger compression causes alignment to disappear. The authors suggest that low-variance directions mainly carry modality-specific or noise information, while shared structure is distributed across multiple principal components.
• Appendix C: on DOCCI, more variance must be retained to maintain alignment, but language representations compressed to 100 dimensions are still largely maintained (Figure 8); COCO results are in Figure 9.

Capability-Alignment Correlation

• What was measured: correlation between MMLU/ARC/GSM8K scores of 21 generative language models and alignment with DINOv3 (Figures 3, 4).
• Results (maximum alignment layer): on DOCCI, MMLU r = 0.66 (p = 0.001), still r = 0.66 after calibration (p = 0.001); when DOCCI is summarized to 10%, r = 0.59 (p = 0.005); WIT r = 0.46 (p = 0.03); CC3M r = 0.26 (p = 0.25). ARC on DOCCI r = 0.73 (p = 0.0002), unchanged after calibration; GSM8K has no statistically significant correlation with alignment.
• Authors' interpretation: caption detail itself cannot explain the trend; sufficiently clean and visually informative data matters more; cross-modal alignment is not a general proxy for language model capability and may be related only to capabilities that depend on semantic knowledge.

Baseline Contextualization

• What was measured: comparison of alignment across four regimes: randomly initialized, unimodal model pairs, VLMs, and same-modality model pairs (Figure 5, 3 runs).
• Results: unimodal model pairs exceed 0.3 mutual k-NN on COCO and DOCCI, higher than CLIP and SigLIP 2, and lower than Qwen3-VL-Embedding-8B; same-modality model pairs are approximately 0.5. VLM projection-layer scores in Table 1: CLIP on COCO is 0.317 ± 0.005, SigLIP 2 is 0.285 ± 0.004, Q3-VL on DOCCI is 0.445 ± 0.003.
• Authors' interpretation: unimodal models learn partially shared representations, but the gap compared with same-modality models remains large; shortening DOCCI captions lowers absolute alignment but does not change the relative ordering (Appendix F).

Other Ablations and Analyses

• k and N (Appendix G, DOCCI): for k ∈ {1, 5, 10, 100}, scores increase as k increases, but the regime ordering is unchanged; with fixed k/N ≈ 0.01, scores are relatively stable.
• VLM layer choice (Appendix E, Table 1): the maximum score difference between projection layer and maximum layer pair is 0.016 (SigLIP 2, COCO), and the rest do not exceed 0.006.
• ARC/GSM8K on other datasets (Appendix D): the positive correlation for ARC is weakly visible on WIT and summarized DOCCI, and there is almost no consistent relationship on CC3M; GSM8K has no consistent positive correlation on any dataset; final-layer alignment is generally less predictive.

Limitations and Future Directions Noted by the Authors

• Scope of applicability of the conclusions (Conclusion/Limitations): the conclusions are limited to the evaluated models, datasets, and alignment metric.
• Generalization to other modalities (Limitations): the paper studies only cross-modal alignment between unimodal encoders in the vision and language domains; whether it generalizes to other modalities remains open.
• Singularity of the metric (Limitations): mutual k-NN characterizes only one kind of representational alignment, and its absolute scores depend on metric hyperparameters, representation extraction choices, and evaluation data; the authors mitigate but do not eliminate this dependence using the calibrated variant and baseline comparisons.

Scope of Experimental Coverage

• Models evaluated: 3 vision models, 21 generative language models, 2 text embedding models, 3 VLM references, and randomly initialized models (Section 2).
• Datasets: WIT, CC3M, COCO, DOCCI, and 4 summarization ratios of DOCCI; N = 1024 image-text pairs and k = 10 per setting (Section 2).
• Capability benchmarks: MMLU, ARC Challenge, GSM8K, evaluated via the LM Evaluation Harness (Section 2).
• Number of repetitions: 5 runs in Section 3.2, 3 runs for Figure 5 and Table 1, and a single run for WIT (Section 3.2, Appendix E).
• Not reported in the paper: a complete list of specific version numbers and sizes for each language model (the main text lists only family names, and the scatter plots label some models); a complete list of the number of layers and hidden dimensions for each model in the layer-wise analysis.

An analytical paper that re-examines whether the Platonic Representation Hypothesis (PRH) holds for the latest models, where alignment appears, and how the scores should be interpreted.

• Problem: PRH holds that vision and language models converge to similar representations, but recent work questions whether the performance-alignment trend still holds for new models, and similarity scores lack baselines and are hard to interpret; existing studies report only the maximum layer-pair similarity.
• Method: On WIT, CC3M, COCO, and DOCCI, using mutual k-NN and its calibrated variants, analyze layer-wise alignment between 3 vision models and 23 language models, alignment under PCA compression, and correlation with MMLU/ARC/GSM8K scores, and compare against randomly initialized models, explicitly aligned VLMs, and same-modality model pairs.
• Locating alignment: Cross-modal alignment is mainly in later layers, peaking at 0.143 on WIT (DINOv3 layer 21 × Qwen3-8B layer 30); the final layer never reaches maximum alignment; alignment is preserved within a subspace of about 100 dimensions (Figure 1, 2).
• Capability-alignment: On DOCCI, MMLU correlates with alignment at r = 0.66 (p = 0.001) and ARC at r = 0.73 (p = 0.0002), and the trend holds at 10% summary (r = 0.59); weaker on WIT and CC3M; GSM8K is uncorrelated. The authors argue that clean and informative captions are key to the trend holding, and that alignment is not a general proxy for capability.
• Baseline comparison: Same-modality model pairs exceed 0.3 alignment on COCO/DOCCI, higher than CLIP and SigLIP 2, lower than Qwen3-VL-Embedding-8B, and far below the roughly 0.5 of same-modality models (Figure 5).
• Authors' conclusion: Cross-modal alignment is localized in both depth and representation space; the PRH performance-alignment trend still holds for new models when captions are sufficiently informative; same-modality models can reach alignment at the level of explicitly aligned VLMs, but the gap with same-modality models remains large. Conclusions are limited to the evaluated models, datasets, and metrics.