FragileFlow: Constraining Correct yet Fragile Predictions in Foundation Models via Spectral Control

FragileFlow is a plug-and-play regularizer that uses a calibrated margin buffer to identify fragile samples in LLMs and VLMs that are correctly predicted yet whose probability mass has flowed to the wrong class, and organizes them into a class-wise vulnerable-risk matrix. The paper gives the first PAC-Bayes upper bound for this margin-aware error flow, and states that empirical spectral control can conservatively achieve deterministic worst-class robustness under stability conditions. In multiple-choice LLM benchmarks and few-shot CLIP adaptation, its theoretical risk measure outperforms matched baselines, and in most settings it improves worst-class accuracy after perturbation while maintaining clean accuracy.