Researchers propose TERM, an inference-time jailbreak defense method that requires no preset safety direction

Core question: in long-context prefilling, how can block sparse attention select which KV blocks to keep without losing output accuracy.

• Setting and motivation: as context grows longer, the quadratic cost of self-attention makes prefilling the main bottleneck in LLM inference; block sparse attention performs exact attention only on important blocks and skips the rest, but the difficulty is identifying important blocks without computing the full attention matrix.
• Shortcomings of existing methods (as stated in the paper): training-free methods mostly use attention mass to estimate a block's contribution to the full-attention output, and STEM further adds the value norm; the authors argue that "a block's contribution to the full-attention output" is not equal to "the output shift caused by removing that block," because after a block is removed, softmax re-normalization amplifies the contribution of retained blocks and compensates for part of the removed contribution.
• Core observation: the authors propose output deviation—the distance between the block output and the full-attention output; the authors report that blocks with similar attention mass can differ in output deviation by more than 20×, showing two modes, low deviation and high deviation (Figure 1), so a proxy based only on contribution cannot distinguish blocks that are "safe to remove" from "critical" blocks.
• Method proposed in the paper: POEM (Prefilling via Output Error Minimization), a training-free block sparse attention method: it selects blocks using an output-relative score that combines attention mass and output deviation, and derives a predicted divergence upper bound weighted by head sensitivity, on which basis it calibrates retention thresholds offline for each head group.
• Target evidence form: maintain accuracy close to full attention on LongBench and RULER while achieving up to 6.5× attention speedup relative to FlashAttention-2 (as stated by the authors).

Training-free sparse attention

• SALE, Sketch&Walk, UniSparse, Prism: score blocks using low-bit estimation, cross-layer random sketch propagation, multi-granularity compression routing, and spectrum-aware scores, respectively, to reduce selector cost.
• SpargeAttention, Pbs-Attn, ProxyAttn: reduce cost in other stages—co-designing the selector and kernel, packing important tokens into fewer blocks, and having a few representative heads score for the remaining heads.
• MInference, FlexPrefill, XAttention, STEM: mainly rank and select blocks by attention mass; STEM additionally adds the maximum value norm within a block to make the proxy value-aware. The paper points out that these methods all ignore the re-normalization effect and do not consider output deviation, which is the direct difference from POEM (Table 1).

Training-based sparse attention

• Native Sparse Attention, MoBA, InfLLM-V2, HiLS: learn sparse patterns during pretraining or adaptation, such as a combination of compression + selection + local attention, top-k block routing, switching between dense/sparse by sequence length, and learning hierarchical retrieval from the language modeling loss.
• SeerAttention / SeerAttention-R, DeepSeek Sparse Attention, SAS: train block-level gating on a frozen backbone (prefilling and decoding), or use a token-level indexer or language modeling loss for context ranking. The paper points out that these methods all require training the model or an auxiliary selector, whereas POEM is training-free and can be applied directly to pretrained models.

Baselines and benchmarks

• Baseline methods: dense FlashAttention-2, and four training-free sparse prefilling baselines, MInference, FlexPrefill, XAttention, and STEM, all using their released or recommended configurations without task-specific tuning; SpargeAttention is excluded, with the paper's stated reason being that its online PV skipping does not fall within the scope of block selection comparison on the same shared backend.
• Benchmarks/datasets: LongBench (21 tasks, 6 categories), RULER (six lengths from 4K to 128K), PG19 (perplexity ablation and diagnostics).

The authors position POEM as a method in training-free block sparse attention that selects blocks by "the output error caused by removal": combining attention mass and output deviation, rather than only estimating a block's contribution to the full-attention output.

Overall idea: change "which blocks to select" from "largest contribution" to "smallest output error after removal". POEM consists of three parts: output-relative block selection, hierarchical error decomposition, and offline threshold calibration; the pipeline is shown in Figure 2 and Algorithm 1.

Output-relative block selection (Section 4.1)

• Objective: select a block set Si that minimizes the error between the sparse output and the full-attention output; the authors write min_Si |o_i − õ_i|₂, obtain an upper bound via the triangle inequality, and then define each block's importance as Iib = mib·‖v̄ib − oi‖2, i.e., the product of attention mass and output deviation.
• Practical score: take the logarithm of Iib and approximate the block output with a first-order Taylor expansion, yielding the selection score s_ib = (q̄_i k̄_b^⊤)/(√d_h) + β log |v̄_b + (q̄_i)/(√d_h) C_b − ô_i|₂, where Cb is the intra-block key–value covariance, the first term follows XAttention's antidiagonal scoring, and β adjusts the strength of the deviation term.
• Two-stage selection: Stage 1 coarsely selects a candidate pool by attention mass using a conservative threshold p1 ≥ p, and constructs a coarse output reference ôi; Stage 2 reranks within the candidate pool by the output-relative score, using threshold p2 = p/p1 to retain nominal mass p. Main configuration p = 0.9, p1 = 0.94.

Hierarchical error decomposition (Section 4.2)

• Prediction to layer: the objective is written as minimizing the KL divergence between the dense and sparse predictive distributions; a first-order expansion of the downstream network gives the KL upper bound dKL ≤ cℓ·‖∆Oℓ‖²F (dominant term), where cℓ is composed of the softmax Fisher matrix and the spectral norm of the layer-to-logits Jacobian, measuring the sensitivity of the prediction to that layer's output.
• Layer to head: the layer output is the projected sum of the head outputs; the authors use the triangle and Cauchy–Schwarz inequalities to decompose the layer error bound into a per-head error sum weighted by the projection norm ‖WO^(h)‖²₂, which leads to two components: output-relative selection reduces per-head error, and threshold calibration handles differences in per-head sensitivity.

Offline threshold calibration (Section 4.3 and Appendix F)

• Per-head-group thresholds: replace the shared p2 with per-head-group thresholds p2,h; following GQA's KV grouping, heads sharing a KV projection share one threshold.
• Error-ranked batch bisection: first run a single-group perturbation probe on the ranking set Drank (multiply that group's threshold by 0.9 while keeping the others at 0.9), and sort the groups in ascending order of KL error between the final logits and the dense reference; then try lowering thresholds in batches on the search set Dsearch, accepting the whole batch if the score does not decrease, otherwise bisecting and first retrying the half with lower error; stop when the maximum number of steps Tmax is reached or all groups have been processed. In the end each group needs only 1 bit of record (not lowered 0.90/0.94 or lowered 0.81/0.94).

Implementation and key parameters (Section 5, Appendix C.5)

• Scoring and kernel: Stage 1 uses low-precision (INT8) Q/K copies to estimate block attention mass, with antidiagonal scoring B1 = 16, and Stage 2 uses B2 = 8; after block selection, FlashAttention-2 computes exact attention on the retained blocks.
• Fixed retained blocks: following XAttention/STEM, sink and local blocks are retained—4 each for Llama-3.1-8B, and 1 each for Qwen3-8B/14B.
• Hyperparameters: β = 0.3 (Qwen3-14B, Llama-3.1-8B), β = 0.2 (Qwen3-8B); the Qwen series uses YaRN to extend the context window to 128K; log lower bound 10⁻⁶; hardware is a single NVIDIA A100.

Experimental Setup

• Models tested: Llama-3.1-8B, Qwen3-8B, Qwen3-14B, with weights frozen; the Qwen series is extended to 128K using YaRN.
• Baselines: Dense FlashAttention-2 and MInference, FlexPrefill, XAttention, STEM, all using released or recommended configurations and the same FlashAttention-2 backend.
• Benchmarks: LongBench (21 tasks in 6 categories, reporting scores for each category and the unweighted macro average), RULER (4K/8K/16K/32K/64K/128K, equally weighted average), PG19 (perplexity ablation).
• Metrics: task scores, Word PPL, attention latency (kernel/total), MSE of layer outputs and logits, cosine similarity, candidate pool miss rate.
• Main configuration: p = 0.9, p1 = 0.94; β = 0.3 (Llama-3.1-8B, Qwen3-14B), 0.2 (Qwen3-8B); 128-token blocks.
• Sample sizes (where stated in the paper): Table 2 uses 30 LongBench texts of 8,192 tokens; Table 4 uses 20 books from the PG19 test set with 32K input; Appendix D diagnostics use 30 LongBench texts or 2 PG19 texts. The sample size and number of repetitions for each task in the main table (Table 3) are not reported in the paper.

Main Results

According to Table 3 (LongBench and RULER average scores; RULER is the average over six lengths from 4K to 128K):

Swipe horizontally to read the table

Model	Method	LongBench Avg.	RULER Avg.
Llama-3.1-8B	Dense	40.38	88.97
Llama-3.1-8B	POEM	40.16	87.74
Llama-3.1-8B	Best baseline (MInference / FlexPrefill)	39.78	87.71
Qwen3-8B	Dense	31.56	86.52
Qwen3-8B	POEM	30.83	84.94
Qwen3-8B	Best baseline (STEM / XAttention)	30.73	84.59
Qwen3-14B	Dense	38.77	91.72
Qwen3-14B	POEM	38.68	89.68
Qwen3-14B	Best baseline (STEM / MInference)	38.24	89.63

• The authors state that POEM ranks first among sparse methods on LongBench for all three models, with the smallest gap to dense being 0.09 points (Qwen3-14B), and that its average score on RULER is also the highest among the three models.
• Exceptions in the table: on the RULER average for Llama-3.1-8B, FlexPrefill is 87.71 and POEM is 87.74, a gap of 0.03 points; on individual tasks POEM is not first across the board (e.g., for Llama-3.1-8B, S-Doc and M-Doc are lower than MInference).
• Efficiency (Figure 5, Qwen3-8B): at 128K, POEM's total attention latency is 143.9 ms, versus 928.0 ms for dense FA2, which the authors describe as a 6.5× speedup; at each length from 16K to 128K, POEM is the fastest of the six methods. Figure 6 shows that beyond 64K, the sparse attention kernel accounts for about 80% of total time, with the two-stage selection overhead accounting for a small share.

Validity Diagnostics of Output-Relative Scores

• Reconstruction error (Table 2, Qwen3-8B, 30 LongBench 8K texts, fixed 16-block budget): POEM's MSE at layers 5/15/35 (3.25×10⁻⁴, 1.22×10⁻², 5.79) and final logits MSE (2.74) are all lower than mass-only (3.27, 1.23, 5.88, 2.86) and STEM-style scoring (3.27, 1.23, 5.88, 2.85).
• Exact-quantity diagnostics (Figure 3, Llama-3.1-8B-Instruct, RULER samples): the output deviation distribution shows two modes; output-relative re-ranking changes the selected block set for more than 74% of heads under the same retention setting.
• Module ablation (Table 5, Llama-3.1-8B, RULER 128K): the mass-only baseline has Overall Avg. 69.94 (budget 19.50%); adding output deviation raises it to 72.14 (19.56%); adding threshold calibration gives 72.38 with the budget dropping to 18.53%. The authors state that the calibration exploits differences in head sensitivity.

Approximation and Coverage Diagnostics (Appendix D)

• Block output approximation (Figure 7): the weighted median cosine similarity between the first-order estimate and the exact block output exceeds 0.9 in 31 of Llama's 32 layers and 29 of Qwen's 36 layers; exceptions are Llama layer 1 (0.805) and several early Qwen layers and layer 34.
• Global output reference (Figure 8): the median pooled cosine per layer ranges from 0.399 to 0.873 for Llama and 0.510 to 0.914 for Qwen, with some heads negative; the authors state that this reference is only an approximate prior.
• Candidate pool coverage (Figure 9, Llama-3.1-8B, 2 PG19 texts, diagnostic setting with β = 0.6): at p1 = 0.94, the miss rate is at most 0.035%, i.e., covering at least 99.96% of the blocks selected by the fully visible reference; the authors note that results on two texts do not guarantee zero misses on other inputs.

Other Ablations and Analyses

• β sweep (Figure 4a): Qwen3-14B and Llama-3.1-8B perform best at β = 0.3, and Qwen3-8B at 0.2; larger values degrade performance.
• Total threshold p (Figure 4b, Llama-3.1-8B, RULER): gains rise from p = 0.8 to 0.9 and fall at p = 0.95; the authors state that 0.9 is the best trade-off among the tested settings.
• Stage 1 threshold p1 (Figure 4c, Llama-3.1-8B, PG19): perplexity decreases as p1 increases from 0.90 to 0.94, and worsens beyond 0.94.
• Scoring proxy comparison (Figure 4d, Llama-3.1-8B, PG19): at matched density, POEM's perplexity is lower than mass-only and STEM.
• Component ablation (Table 4, Qwen3-8B, 20 PG19 books, 32K, density about 0.1): POEM word perplexity is 43.65; removing covariance correction gives 44.01 (+0.36); removing the output reference gives 43.95 (+0.30).
• Memory (Tables 6, 7): at 128K, auxiliary tensors (INT8 Q/K copies + BF16 covariance) total 896 MiB; end-to-end peak memory for POEM is only 2.005 MiB higher than dense.

Limitations and future directions noted by the authors

• More accurate block output estimation (Conclusion): the authors state that future work could estimate block outputs more accurately before attention computation.
• Extension to decoding-phase KV cache eviction (Conclusion): the authors state that the output-relative criterion could be extended to KV cache eviction during decoding.
• Extension to trainable sparse attention (Conclusion): the authors state that the criterion could be extended to trainable sparse attention.
• Deployment risk warning (Ethics Statement): the authors point out that POEM is an approximation of dense attention and may change outputs, increase factual errors, or amplify biases already present in the underlying model and data in ways not covered by aggregate benchmark scores; they do not recommend using it in safety-critical or high-risk scenarios without application-specific evaluation and safeguards.

Experimental coverage

• Models tested: 3 in total—Llama-3.1-8B, Qwen3-8B, Qwen3-14B—all with frozen weights; the Qwen series is extended to 128K via YaRN (Appendix C.4).
• Benchmarks and tasks: LongBench (21 tasks, 6 categories), RULER (six lengths from 4K–128K), PG19 (perplexity ablation and diagnostics); efficiency evaluation is done on a single NVIDIA A100 with sequences from 16K–128K.
• Baselines: dense FlashAttention-2 and four training-free baselines—MInference, FlexPrefill, XAttention, STEM; SpargeAttention is excluded, and the paper states the reason is that its online PV skipping falls outside the scope of the shared-backend block-selection comparison (Section 5).
• Diagnostic scale: Table 2 uses 30 texts of 8,192 tokens; Table 4 uses 20 PG19 books; the coverage diagnostic in Appendix D uses 2 PG19 texts, 4 layers, and lengths from 16K–64K, with β = 0.6 rather than the model-specific coefficients of the main benchmark (Appendix D.3).
• Unreported information: the paper does not state the sample size and number of repetitions per task in the main results table (Table 3), the sizes of Drank/Dsearch used for threshold calibration, or whether statistical significance tests were performed; the paper notes that the sample identifiers and random seeds of the calibration list will be released with the configuration (Appendix F, authors' plan).

A paper on accelerating long-context prefilling that proposes a training-free block sparse attention method, POEM.

• Problem: block sparse attention relies on importance proxies to select which blocks to keep; existing proxies estimate a block's contribution to the full-attention output. The authors argue that softmax re-normalization amplifies the contribution of retained blocks, so "contribution" is not equal to "the output shift caused by removal." The authors report that blocks with similar attention mass can differ in output deviation by more than 20× (Figure 1).
• Method: POEM defines block importance as the product of attention mass and output deviation, using a first-order Taylor approximation of the block output to obtain a computable output-relative score; the two-stage process first coarsely selects a candidate pool by mass (p1 = 0.94) and then re-ranks within the pool (total target p = 0.9); it also derives a predicted KL upper bound weighted by head sensitivity, and on that basis calibrates retention thresholds offline for each KV head group (a batched binary search over error rankings).
• Main results (Table 3): on Llama-3.1-8B, Qwen3-8B, and Qwen3-14B, POEM's LongBench averages are 40.16/30.83/38.68 and RULER averages are 87.74/84.94/89.68, the highest among sparse methods and close to dense (40.38/31.56/38.77 and 88.97/86.52/91.72).
• Efficiency (Figure 5, Qwen3-8B): at 128K, total attention latency is 143.9 ms versus 928.0 ms for dense FA2, which the authors describe as a 6.5× speedup; it is the fastest of the six methods at each length from 16K to 128K; end-to-end peak memory is 2.005 MiB higher than dense (Table 7).
• Ablations and diagnostics: on RULER 128K, the mass-only baseline is 69.94, adding output deviation raises it to 72.14, and adding threshold calibration gives 72.38 while the budget drops from 19.56% to 18.53% (Table 5); output-relative re-ranking changes block selection for more than 74% of heads (Figure 3d).
• Authors' conclusion: output error depends on both the attention mass of the removed block and its output deviation, and scoring the two jointly can maintain accuracy at high sparsity; the authors state that future work could estimate block outputs more accurately and extend the criterion to decoding-time KV cache eviction and trainable sparse attention.