SimVerity: When Does Agent Success in Simulated Environments Carry Over to Physical Deployment?

The researchers propose SimVerity, an acceptance-transfer framework that replays matched scenarios in real smart-home deployments to test whether agents that pass in simulated environments still succeed in physical deployment. In one execution, task completion, reported status, observable effects, and final settlement results diverge: an advanced simulator passed all 240 lighting trials, but the camera captured 42 sub-second failures that the final-state check did not detect. Erroneous passes can be predicted: a risk profile learned from real-world trials and locked before evaluation can predict failures on paths it has never physically measured, and outperforms attribute-blind baselines in all 11 held-out sessions across two cohorts. Auditability is also measurable: adjusting one agent loop's model client and service configuration raised its scenario match rate from 52-88% to 100%. A second qualified simulator provided no independent cross-validation; it raised no objection on any overlapping case, and only physical measurement exposed the shared blind spot of both.