Skip to content
Failure-First·Indexed by this site · Originally published· Evaluation →· Its place in the scan record →

Bomatter and Gouk benchmark: EEG discriminative OOD detection is near random, generative is effective

[Daily Paper] OOD Detection for EEG-based Machine Learning in High-Risk Environments

AI summary

Bomatter and Gouk's benchmark study evaluated OOD detection methods for clinical EEG models and found that discriminative detectors are close to random under realistic covariate shift (AUROC≈0.50), while generative latent-space methods such as SITN lead by a wide margin. The study applied five types of controlled perturbations on TUAB (over 2,300 people) and CAUEEG (1,122 people): sampling-rate shift, channel permutation, re-referencing, and low-pass and high-pass filtering. On TUAB, SITN reached an AUROC of 0.998 under the Cz re-referencing shift, while MSP reached only 0.517. MSP also exhibited overconfidence anomalies: on mixed data under the TUAB 200 Hz resampling shift, the accuracy of its highest-confidence 0.1% interval fell to 47.0%. Based on this, the paper proposes a two-stage abstention pipeline: first use SITN to filter unfamiliar data, then use calibrated MSP to assess uncertainty, and refer cases that fail the threshold to a human neurologist; on a 50/50 mixed test pool under the 200 Hz shift, it raised accuracy from 67.5% to 84.1% at 43.7% coverage.

Read originalfailurefirst.org