Kierston and KierstonBench: Simulating Allentown, Pennsylvania and Evaluating Language Model Performance on Criminal Investigation and Surveillance Tasks

Researchers released Kierston, a city-scale simulation engine, and its accompanying benchmark KierstonBench, to evaluate large language models' capabilities in criminal investigation and mass surveillance tasks under known simulated events. Kierston simulates the daily activities of approximately 125,000 residents of Allentown, Pennsylvania, generating telecommunications, financial, administrative and license plate records that derive from the same behavioral timeline and are distributed across data sources simulating different providers; models conduct investigations through restricted record interfaces and cannot access the underlying simulation. The benchmark contains 32 task families, covering record linkage, routine reconstruction, incident investigation and screening for deceptive accounts. Evaluation of 16 models showed that most models could not complete record investigations and submitted identifications unsupported by the records; a few frontier models completed most tasks and abstained when evidence was insufficient, but were still misled by common deception and assumptions, and no model could replace a human investigator.