FAR.AI research on scaling trends in SOLiD lie-detector supervision

FAR.AI tests on the SOLiD protocol whether lie-detector-based preference learning supervision scales with model size. SOLiD uses a linear probe to read the model's middle-layer residual stream, flags suspicious answers and escalates them to trusted labelers, and leaves the rest to cheap labelers. On Llama-3 (1B to 405B) and Qwen-3 (0.6B to 32B), when the probe's true positive rate is at least 0.8, the proportion of undetected deception decreases significantly with scale (p < 0.01); at a TPR of 0.99, Llama's missed deception rate drops from 34% at 1B to 14% at 405B. The SOLiD-Defer variant without trusted labelers performs close to the full protocol at low false positive rates. However, the method relies on the probe's training data matching the fine-tuning data distribution; cross-dataset transfer (MASK, TrueFalseFacts to DolusChat) substantially raises false positive rates, especially severe on Qwen.