ByteDance Seed found that DeepSeek-V4 retrieval performance fluctuates periodically with token position

ByteDance's Seed team found that the DeepSeek-V4 series' accuracy in long-context retrieval fluctuates periodically with the position of the target information in the input, and attributed the cause to its chunked KV cache compression. In a needle-in-a-haystack test with 128K tokens and about 16,000 key-value pairs, the maximum accuracy gap between positions for DeepSeek-V4-Flash-Base reached 40.2 percentage points, and for DeepSeek-V4-Pro-Base it was 34.8 percentage points; after post-training the gap narrowed, with DeepSeek-V4-Flash-0731 dropping to 19.1 percentage points, DeepSeek-V4-Pro-0813 dropping to 14.8 percentage points, and DeepSeek-V4.1-Flash-0910 further dropping to 6.1 percentage points, but the periodic variation still remained. The team therefore recommended that when evaluating models with chunked KV cache compression, tests should be run separately across different compression phases rather than only looking at the overall average score.