FAR.AI researchers say they built a dataset of 1,264 fully synthetic scenarios to test a false-statement detection probe.
FAR.AI researchers built the fully synthetic dataset SynthDeception, using 1,264 end-to-end generated realistic agent scenarios to test whether a fully synthetic data pipeline can train a false-statement detection probe. Such probes read the model's internal activations and flag when the model says something it believes to be false. The dataset contains 15,168 follow-up question-answer pairs, with 6 questions per scenario, each question paired with two mirrored answers, one honest and one dishonest, each accounting for half. Scenarios vary along dimensions including deception type, non-deceptive behaviors easily confused with deception, industry domain (including 17 domains such as healthcare and cybersecurity), tool use (about half of the scenarios involve 193 tools), user persona (from Nemotron Personas USA) and context length (about 1k to 100k tokens); the follow-up questions control question focus, question type, presuppositions and answer design to rule out surface cues such as position, style and length.
Can you train falsehood detectors with a fully synthetic data pipeline?
http://FAR.AI researchers built SynthDeception, 1,264 realistic agentic scenarios generated end to end, to test that for probes, which read a model's internal activations to flag when it says something it believes is false. 1/8