Large Language Models Do Not Know What They Know: A Temporal Self-Knowledge Benchmark for Financial Backtesting

The researchers propose the FinDendro benchmark, which uses 2,226 dated US facts from 1999 to 2026 (monthly macroeconomic series, month-end stock prices, Federal Reserve decisions) to test large language models' awareness of the temporal boundaries of their own knowledge, with persistence and dominance-outcome controls holding random accuracy at about 0.27. Results show that most frontier models underestimate their knowledge in self-reports: GPT-5 answers 76% of control facts correctly but claims to know only 38%, and still answers 61% of the facts it claims not to know correctly; knowledge claims can exceed the measured knowledge boundary by up to seven quarters, while stating historical dates eliminates most prior knowledge claims with accuracy unchanged. The authors therefore point out that financial backtests within a model's training window cannot be proven leak-free by self-reports, instructions or vendor-stated knowledge cutoffs, and that FinDendro provides a benchmark and procedure for selecting backtest periods based on measured factual knowledge.