Tight PAC-Bayes Generalization Guarantees for LLM Judges
Tight PAC-Bayes Generalisation Guarantees for Large Language Model Judges
Researchers propose using a PAC-Bayes framework to certify LLM judges that monitor LLM agent outputs, defining risk over their safety classification on prompt-response pairs produced by a deployed generator; through tighter bounds and more efficient coding, standard parameter-efficient fine-tuning suffices to obtain non-vacuous or even tight generalization certificates on classification and Brier risk, without held-out validation data. The authors propose LoRA-GT, which shares a single quantization scale across layers, achieving the tightest certificates with a small drop in predictive performance. Experiments on five guardrail models and four benchmarks show that certificate tightness depends more on adaptation complexity than on base model scale. The paper also examines how far quantization can be pushed before risk increases outweigh the bound tightening from shorter descriptions, and the authors describe these results as a practical path toward certifiable scalable oversight.