ProbeLLM: Automated principled diagnosis of LLM failures

Researchers propose ProbeLLM, a benchmark-agnostic automated probing framework for consolidating scattered large language model failure cases into structured failure modes. The framework models probing as hierarchical Monte Carlo tree search, allocating a limited probing budget between global exploration of new failure regions and local refinement of recurring error patterns. It restricts probing to verifiable test cases, uses tool-augmented generation and verification to provide evidence for failure discovery, and then consolidates discovered failures into interpretable failure modes through failure-aware embeddings and boundary-aware induction. The authors state that across multiple benchmarks and LLMs, ProbeLLM presents a broader, clearer, and finer-grained picture of failures than static benchmarks and existing automated methods, pointing toward a shift from case-centric evaluation to principled weakness discovery. Paper number arXiv:2602.12966.