Skip to content
arXiv:对齐、欺骗与监督· arXiv:2609.25938· Jiamiao Liu·Indexed by this site · Originally published

Which Oracle for certification? Execution labels determine the risk reported by text-to-SQL conformal abstention

Certified Against Which Oracle? Execution Labels Set the Reported Risk of Conformal Abstention for Text-to-SQL

AI summary

For text-to-SQL conformal abstention certificates, their validity depends on the correctness labels used for calibration. The study conducts a pre-registered intervention on Spider-Realistic, replacing the benchmark's own database with its distilled multi-instance test suite. Across four SQL-specific checkpoints and two split schemes, the certificate's held-out risk is 2.73 to 10.23 percentage points higher than reported by its own labels; under blind labeling by two SQL experts, the nominal 0.10 certificate has actual risk of 20.0 and 17.2 percentage points on two checkpoints. The stricter oracle fails in both directions: most of its rejected answers are not errors, and some accepted answers contain errors.

Read originalarxiv.org