MULI: Free toxic prompt detection using the LLM's own logits

The MULI method directly leverages the LLM's own information to detect toxic prompts, outperforming SOTA toxicity detectors on multiple metrics. The study finds that benign and toxic prompts can be distinguished based on the logits distribution of the model's first response token, and accordingly builds a robust detector using sparse logistic regression. The method targets the problems that LLMs may fail to refuse toxic prompts or over-refuse benign prompts, and that existing detectors have low true positive rates at low false positive rates.