ByteDance research reveals the cause of DeepSeek-V4 retrieval fluctuations
Start here
ByteDance's Seed team found that the accuracy of the DeepSeek-V4 series in long-context retrieval fluctuates periodically with the position of the target information in the input, and attributed the cause to its chunked KV Cache compression. In a needle-in-a-haystack test with 128K tokens and about 16,000 key-value pairs, the maximum accuracy gap between different positions for DeepSeek-V4-Flash-Base was 40.2 percentage points, and for DeepSeek-V4-Pro-Base it was 34.8 percentage points; the gap narrowed after post-training.
Generated by AI from reports · Updated 3 days ago
Progress timeline
- 量子位SelectedByteDance Seed found that DeepSeek-V4 retrieval performance fluctuates periodically with token position
ByteDance's Seed team found that the DeepSeek-V4 series' accuracy in long-context retrieval fluctuates periodically with the position of the target information in the input, and attributed the cause to its chunked KV cache compression. In a needle-in-a-haystack test with 128K tokens and about 16,000 key-value pairs, the maximum accuracy gap between positions for DeepSeek-V4-Flash-Base reached 40.2 percentage points, and for DeepSeek-V4-Pro-Base it was 34.8 percentage points; after post-training the gap narrowed, with DeepSeek-V4-Flash-0731 dropping to 19.1 percentage points, DeepSeek-V4-Pro-0813 dropping to 14.8 percentage points, and DeepSeek-V4.1-Flash-0910 further dropping to 6.1 percentage points, but the periodic variation still remained. The team therefore recommended that when evaluating models with chunked KV cache compression, tests should be run separately across different compression phases rather than only looking at the overall average score.
Related discussion
- Older than 72 hours · Not included in the 72-hour figures
- 网页IT之家字节 Seed 团队发现 DeepSeek“抽风”原因，长上下文可能性能漂移(opens the original post in a new tab)
Attention trend
New sources per day
- Oct 9New2 sources (2 outlets,0 accounts)
- Oct 10New0 sources (0 outlets,0 accounts)
- Oct 11New0 sources (0 outlets,0 accounts)
- Oct 12New0 sources (0 outlets,0 accounts)