Less is more: the LLM scale paradox in context compression

In compressor–decoder lossy context compression, increasing compressor scale can reduce faithful recovery of the original text even as reconstruction error declines. The authors call this the Size-Fidelity Paradox.

• Scenario: the authors state that the scaling hypothesis dominates LLM training, and larger models usually generate better. Context compression maps natural language into a small number of memory tokens to speed up decoding; existing work also reports better compression quality from larger compressors.
• Limitations of existing evaluation: training loss, perplexity, BLEU, ROUGE, and BERTScore mainly reward fluency and surface similarity. The authors state they cannot distinguish whether content comes from the compressed representation or is filled in by parametric priors, so high reconstruction scores can still alter or drop details of the original text.
• Core observation: beyond a certain parameter scale, larger compressors can be worse than smaller models in fidelity. The authors summarize two dominant factors: knowledge overwriting, replacing source facts with internal beliefs; and semantic drift, paraphrasing or reorganizing rather than verbatim reproduction.
• Approach in this paper: two diagnostic tasks isolate these two failures, tested across 27 compressor settings spanning Qwen-3 and LLaMA-3.2, 0.6B to 90B, at 4×/16×/64×; effective rank and conditional entropy then analyze how compressed memory is organized and how tight the reconstruction constraint is.

Section 2 of the paper is divided into two groups, scaling behavior and long-context compression, with diagnostic data drawn from the knowledge conflict literature.

Scaling laws and their exceptions

• Kaplan et al. (2020), Hoffmann et al. (2022), and subsequent work cited in the paper: increasing scale generally reduces training loss and improves various generation and reasoning performances; larger models often have stronger world knowledge.
• Wei et al. (2023), Zhou et al. (2024), Chen et al. (2025): scale does not guarantee uniformly better results. The paper states that these works report larger models can be less reliable on difficult inputs, such as high-confidence errors or weaker calibration; but they mainly look at correctness in QA or reasoning, offering limited insight into faithful reproduction of inputs.

Long-context compression and reconstruction evaluation

• Ge et al. (2023)'s in-context autoencoder, Mu et al. (2023)'s gist tokens, Chevalier et al. (2023), Dai et al. (2025)'s pretrained context compressors, Li et al. (2025)'s 500xCompressor, etc.: most adopt a compressor–decoder that encodes long sequences into compact latent representations and then conditionally decodes.
• The paper states that prior evaluations are reconstruction-centric, comparing training loss, perplexity, BERTScore, BLEU, and ROUGE by compression ratio and model scale. The authors argue these metrics may only be surface proxies for compression quality.

Knowledge conflict data

• Ming et al. (2024)'s FaithEval, Xie et al. (2024)'s ConflictQA, and Longpre et al. (2021)'s entity knowledge conflicts: examine whether models follow context when context and parametric knowledge conflict. This paper uses the first two to construct counterfactual diagnostics, without treating them as compression algorithms.
• Downstream use of MuSiQue, QASPER, and BBH, looking at compressed retrieval-augmented QA and in-context learning. The paper does not treat the original methods on these datasets as compression baselines.

Baselines and positioning of this paper

• Baseline methods: there is no external method baseline compared against other compression algorithms. The comparison is between compressors of different scales under the same training protocol, and surface reconstruction metrics relative to diagnostic QA. Benchmarks/datasets are FineWeb, FaithEval, ConflictQA, with ArxivCorpus as the cross-domain.
• The authors position this paper in a fidelity-first compressor–decoder setting: the goal is to reconstruct the original source with minimal distortion, not to generate plausible paraphrases, and to analyze how scale changes memory organization and weakens literal recovery.

The authors train multi-scale compressors with the same protocol, freeze the decoder for reconstruction, and then use diagnostic QA and memory probes to see whether fidelity declines with scale.

Compression setup and training objective

Given a vocabulary sequence x, the compressor maps it to continuous memory. The paper's core mapping is Z=f_θ(x), where the number of memory slots M is much smaller than the length L, and the compression ratio is ρ=L/M. The decoder is a causal language model conditioned on Z, estimating the original text probability step by step from the prefix.

The training loss combines reconstruction and continuation: L=L_re+L_nt. L_re requires reconstructing the compressed prefix under Z, and L_nt performs autoregressive prediction on the subsequent segment. The authors state this differs from summarization that allows paraphrasing: any deviation from verbatim input is treated as information loss. In the λ sweep later, λ weights L_nt and (1−λ) weights L_re, with a default of λ=0.5.

Memory configuration and training protocol

• Segments: compression segment length is 256 tokens. Under compression ratio r, the number of memories is m=256/r, so 4×, 16×, and 64× use 64, 16, and 4 memory embeddings, respectively. Learnable memory query tokens are appended at the segment end, and their last-layer hidden states are taken; multiple segments are compressed independently and then concatenated, with memory boundary embeddings added.
• Optimization: AdamW, learning rate 1×10−4, cosine, global batch size 512, warmup 300 steps, gradient clipping 0.5, 1 epoch, bfloat16. The trainable parts are the compressor and memory converter; the decoder is frozen. 32B uses ZeRO-3, and smaller models use ZeRO-1. Models are selected by reconstruction validation loss. Appendix B gives tokenization alignment, adjacent segment pairing, and the rule for discarding documents with fewer than 512 compressor tokens.
• Models: the main decoder is Meta-LLaMA-3.2-8B-Instruct. LLaMA-3.2-11B and 90B have the vision encoder removed, leaving only the text backbone. Training text comes from FineWeb.

Knowledge overwriting diagnostics

• Data: FaithEval and ConflictQA introduce context that contradicts existing facts. The authors keep examples with clear conflicts, so that source evidence and parametric knowledge can be separated.
• Judgment: after each counterfactual context is compressed, a question targeting the modified fact is paired with it. Candidates include the source-consistent counterfactual answer and reasonable world-knowledge alternatives. Knowledge fidelity is the accuracy of selecting the source-consistent answer.
• Materials: Appendix D gives the construction, filtering, and optional candidate generation prompts.

Semantic drift diagnostics

• Question generation: on FineWeb and FaithEval, DeepSeek-R1 generates questions requiring precise relations rather than just topic summaries. The main text lists seven dimensions: main topic, entity list, predicate exactness, relation anchor, coreference, role binding, modifier scope.
• Answer constraints: answerable questions must have answers that are exact substrings of the original text. Table 3 reports micro-averaged QA accuracy across the seven dimensions. Appendix D gives the full prompts, and Appendix F gives per-dimension results.
• Filtering: Qwen3-0.6B, 1.7B, 4B, 8B, and 32B compress and reconstruct at 16×, and Exact Match is computed against the normalized source-consistent gold answer. For overwriting questions, if the EM of all five compressors is greater than 0.8, the question is discarded; for drift questions, the 20 QA pairs per context are first averaged into an EM, and if all five compressors exceed 0.8, that context is discarded.

Memory probes and downstream protocols

• effective rank: singular values are taken from the batch-stacked memory matrix, and the entropy of the normalized singular values is exponentiated, measuring how many directions the representation spans. The authors link this to knowledge overwriting.
• conditional entropy: teacher forcing is performed on the gold prefix, the entropy of the decoder's next-token distribution is computed, then averaged over tokens, denoted H(Z). The authors link this to semantic drift: high entropy means multiple plausible continuations remain competitive. Fixed-scale training dynamics are used to judge whether the compression objective favors high or low values.
• Other protocols: generalization experiments change the decoder, pretraining corpus, or loss weights, while the main compression protocol stays unchanged. Downstream RAG compares no context, original context, and 4× compressed reconstructed context; ICL compares zero-shot, full examples, and a 4× setting where only demonstrations are compressed, with the test query uncompressed.

Experimental setup

• Compressors: Qwen3 0.6B, 1.7B, 4B, 8B, 32B; LLaMA-3.2 1B, 3B, 11B, 90B. Each scale has 4×, 16×, and 64×, which the authors call 27 settings in total.
• Decoder: the main experiments fix Meta-LLaMA-3.2-8B-Instruct. Section 5.1.1 additionally uses Qwen3-0.6B and 4B; Appendix E pairs Qwen3-32B with LLaMA-3.2-8B and Qwen3-32B.
• Scale: Table 1. Reconstruction uses 4,444,672 contexts from FineWeb. Overwriting: FaithEval 598 contexts, 598 QA pairs; ConflictQA 1,356 contexts, 1,356 QA pairs. Drift: FineWeb 449 contexts, 8,980 QA pairs; FaithEval 598 contexts, 11,960 QA pairs. The cross-domain corpus is ArxivCorpus, about 20.9B tokens.
• Metrics: surface metrics are BLEU, ROUGE-L, chrF++, BERTScore-F1. Diagnostic metrics are source-consistent answer accuracy and seven-dimension micro-averaged QA accuracy. Filtering uses Exact Match. The paper does not mention LLM judges, judgment thresholds, or human agreement.
• Downstream: MuSiQue and QASPER report F1 and EM; five BBH subsets report EM, and the paper writes 250/187 evaluation examples per subset. The paper does not state the downstream inference model or report the number of repetitions.

Main results

The table below follows Table 3, compression ratio 4×, decoder Meta-LLaMA-3.2-8B-Instruct. The table indicates the Reconstruction column represents BLEU. Qwen3-1.7B is omitted: BLEU 0.94, overwriting mean 0.83, drift mean 0.82.

Swipe horizontally to read the table

Compressor	BLEU	Overwriting mean	Drift mean
Qwen3-0.6B	0.95	0.83	0.83
Qwen3-4B	0.94	0.84	0.83
Qwen3-8B	0.94	0.83	0.81
Qwen3-32B	0.97	0.74	0.69
LLaMA-3.2-1B	0.94	0.83	0.82
LLaMA-3.2-3B	0.90	0.83	0.82
LLaMA-3.2-11B	0.92	0.73	0.70
LLaMA-3.2-90B	0.97	0.71	0.64

• The authors state the scale effect is non-monotonic, with about 3–4B often more stable. The best within group is not unique: at 4×, Qwen3-0.6B and 4B have the same drift mean of 0.83; LLaMA-3.2-1B and 3B have the same overwriting mean of 0.83. The relative declines of the largest models in the table are Qwen3-32B drift −16.87% and LLaMA-3.2-90B drift −21.95%.
• At 16×, LLaMA-3.2-90B has the highest BLEU in its group at 0.86, overwriting mean 0.65 (relative decline −20.73%), drift mean 0.60 (−27.71%). At the same ratio, Qwen3-32B has BLEU 0.85, lower than 1.7B and 4B's 0.86, overwriting mean 0.67 and drift mean 0.66, lower than 4B's 0.83.
• At 64×, the highest BLEU in each group appears in the largest model: Qwen3-32B 0.29, LLaMA-3.2-90B 0.33. The authors give an example: LLaMA-90B has higher BLEU than LLaMA-3B (0.33 vs. 0.29) but lower overwriting accuracy (0.53 vs. 0.62). At this ratio, the best Qwen3 drift mean within group is 4B and 8B tied at 0.44, with 32B at 0.31.

Decoder, corpus, and loss weights

• Decoder: Table 4, Qwen3 compressors, 16×, Qwen3-4B decoder. The 4B compressor's FaithEval overwriting is 0.66 and ConflictQA 0.81; 8B is 0.62 and 0.75. After switching to the Qwen3-0.6B decoder, reconstruction and QA are overall lower; across the four QA columns, 8B is lowest everywhere, but 4B and 0.6B do not move uniformly in the same direction, with FaithEval overwriting 0.54 vs. 0.59. The authors state decoder capacity modulates the magnitude but does not eliminate the higher error rate of 8B relative to smaller compressors. Appendix Table 12: when Qwen3-32B is paired with Qwen3-32B, FaithEval overwriting is 0.63, ConflictQA 0.74, FineWeb drift 0.55, FaithEval drift 0.70, still lower than Qwen3-4B paired with Qwen3-4B's 0.66, 0.81, 0.63, 0.77. When the decoder changes from LLaMA-3.2-8B to Qwen3-32B, FineWeb drift goes from 0.57 to 0.55 and FaithEval drift from 0.74 to 0.70.
• Cross-domain: Table 5, Qwen3 pretrained at 16× on ArxivCorpus. Reconstruction goes from 0.72, 0.76 to 0.77; FaithEval overwriting is 0.60, 0.69, 0.63, and ConflictQA 0.77, 0.82, 0.79. The authors state fidelity QA remains non-monotonic, with 8B below 4B.
• Loss weights: Table 6 in the text lists Qwen3-0.6B at 4×, and Qwen3-4B with λ=0.3, 0.5, 0.7. At default λ=0.5, 0.6B has FineWeb reconstruction 0.95 and ConflictQA 0.95; at λ=0.9, 0.87 and 0.85. The authors state the ratio shifts the reconstruction–fidelity tradeoff but does not restore fidelity increasing monotonically with scale.

Downstream RAG and ICL

• RAG: Table 7, after 4× compression, reconstruction by the fixed decoder, then QA. MuSiQue no-context F1 9.11, EM 2.67; with original text 33.26 and 23.91; after compression, Qwen3-0.6B is 24.14 and 15.08, 32B is 14.55 and 0.53. On QASPER, 0.6B is 28.93 and 19.16, 32B is 23.94 and 13.18, both below the 46.22 and 31.90 with original text. The authors state 8B and 32B are worse than 0.6B and 1.7B. Exception: 1.7B's QASPER F1 is 29.46, higher than 0.6B's 28.93.
• ICL: Table 8, only demonstrations are compressed. BBH five-subset EM mean: zero-shot 50.0, full ICL 70.4; at 4×, 0.6B is 65.2, 4B is 63.6, 8B is 60.8, 32B is 60.9. The authors state larger compressors bring lower accuracy, especially 8B and 32B. Exception: on Tracking Shuffled Objects, 32B is 57.6, higher than 8B's 54.8 and 4B's 56.8.

Memory geometry and conditional entropy

• Rank: the authors state that in Figure 3(a), effective rank rises with compressor scale; in Figure 3(b), fixed-scale training first expands then contracts; in Figure 3(c), after the peak, higher rank corresponds to lower knowledge-fidelity QA. The main text gives no correlation coefficient. In Table 14, eRank goes from Qwen3-0.6B's 856.8 to 32B's 1169.3, and eRank/d drops from 0.8455 to 0.2284; LLaMA-3.2-90B has eRank 1194.4 and eRank/d 0.1458.
• Truncation: Table 15, Qwen3-8B, 16×. When retaining 90% spectral energy, FineWeb reconstruction is 0.8493 (+0.0074), ConflictQA 0.9104 (+0.0173), FineWeb drift QA 0.7452 (+0.0164). At 75% and 50%, all three are below the untruncated 0.8419, 0.8931, 0.7288. The authors state moderate truncation removes the low-energy tail, while overly strong truncation cuts directions carrying source information.
• Entropy: the authors state that in Figure 4(a), larger compressors have higher conditional entropy than medium scales; in Figure 4(b), fixed-model training decreases entropy; in Figure 4(c), higher conditional entropy corresponds to lower semantic drift QA. The main text gives no entropy values. The text of Appendix Figure 7 states that in the λ sweep at 4× for Qwen3-0.6B and 4B, average entropy is lowest near λ=0.5.

Other ablations and analyses

• Table 2, Qwen3, 4×, FineWeb: BERTScore-F1 from 0.995 to 0.998; the highest BLEU is 32B's 0.968, with 8B at 0.940, below 0.6B's 0.952.
• Figure 2: the authors state larger compressors decrease loss faster and reach lower loss; the main text gives no loss values.
• Section 7 uses Table 9 for qualitative comparison: the authors state overwriting and drift can look like hallucination, but compression errors are failures to preserve source facts or relations, and even if the replacement content is true in the world, it counts as failure.
• Appendix Table 11 gives complete means for the four reconstruction metrics; Table 13 gives per-dimension drift QA. The main conclusion in the text uses the micro-average from Table 3.
• The model selection metric is reconstruction validation loss, not diagnostic QA.

Limitations and future directions noted by the authors

• Conclusions remain tied to the memory-token compressor–decoder and the corresponding training objective. Retrieval-based compression, extractive compression, or architectures with explicit copy mechanisms may exhibit different scaling behavior (Appendix A).
• Effective rank and conditional entropy are diagnostic probes for the compressed memory Z, not a complete causal explanation of the paradox, nor do they by themselves provide a complete intervention to restore fidelity in larger compressors (Appendix A).
• The rank intervention further suggests that fidelity depends on learned structure in memory aligned with the source, not just scalar quantities like rank (Appendix A).
• Diagnostic QA is used to stress source-specific fidelity, especially counterfactual facts and fine-grained semantic relations. The authors state they cannot replace all downstream evaluation; different applications may tolerate paraphrase, abstraction, or factual substitution differently (Appendix A).
• The authors believe compressors should be selected and trained according to the fidelity requirements of the intended use. Explicitly strengthening source anchoring in larger compressors while maintaining efficiency is a future direction (Appendix A).

Experimental coverage

• Compressors are Qwen3 at five scales and LLaMA-3.2 at four scales, with compression rates 4×, 16×, 64×; the main-result decoder is Meta-LLaMA-3.2-8B-Instruct (Section 4, Table 3).
• Decoder configurations also include Qwen3-0.6B, Qwen3-4B, LLaMA-3.2-8B, and Qwen3-32B (Section 5.1.1, Appendix E, Table 4, Table 12).
• The pretraining corpus is FineWeb; there are also Qwen3 compressors at 16× on ArxivCorpus. The loss-weight sweep in the main text corresponds to Qwen3 at 4× (Section 5.1, Table 5, Table 6).
• Downstream tasks are MuSiQue, QASPER, and BBH's Date Understanding, Causal Judgement, Logical Deduction, Formal Fallacies, Tracking Shuffled Objects, at a compression rate of 4× (Section 5.2).
• The paper does not report human agreement, statistical significance tests, or number of repetitions for diagnostic QA, nor does it report the reasoning models used for RAG and ICL.

The authors report a Size-Fidelity Paradox in context compression: when the compressor is larger, reconstruction can be better while fidelity to the original can be worse.

• The setting is compressor–decoder. Long text is compressed into a small number of memory tokens, then required to be recovered verbatim, rather than generating fluent paraphrase. The authors argue that surface metrics like BLEU cannot reveal facts being replaced by priors or relations being rewritten.
• The approach fixes the Meta-LLaMA-3.2-8B-Instruct decoder and trains with the same protocol on Qwen3 and LLaMA-3.2, from 0.6B to 90B, at 4×/16×/64×. Counterfactual QA tests knowledge overwriting, and seven-dimensional relational QA tests semantic drift; effective rank and teacher-forced conditional entropy are used to see whether memory becomes broader and reconstruction more uncertain.
• In the 4× results of Table 3, LLaMA-3.2-90B has a BLEU of 0.97 and a mean drift of 0.64; Qwen3-32B has a BLEU of 0.97 and a mean drift of 0.69. In the same table, the 3–4B tier has mean drift between 0.82 and 0.83.
• At 64×, LLaMA-90B has a BLEU of 0.33 and a mean overwriting of 0.53, while LLaMA-3B has 0.29 and 0.62. At 16×, LLaMA-90B's drift decrease relative to the best in group is −27.71%. Reconstruction does not rise with scale at every step: at 16×, Qwen3-32B has a BLEU of 0.85, lower than 4B's 0.86.
• When downstream tasks are run after 4× compression, MuSiQue F1 goes from 24.14% for Qwen3-0.6B to 14.55% for 32B (Table 7). BBH five-subset EM mean goes from 65.2% for 0.6B to 60.9% for 32B (Table 8), and on Tracking Shuffled Objects, 32B is higher than 4B and 8B.
• The authors conclude that properties that favor complex reasoning also weaken the rigid constraints needed for faithful reconstruction; scaling laws do not transfer directly when the objective shifts from plausible generation to faithful preservation. Compressor scale should be considered together with the fidelity requirements of the application.