Invariant-aware Harness enforcement protects LLM agents from silent API failures
Invariant-Aware Harness Enforcement to Protect LLM Agents from Silent API Failures
Researchers propose SilentGuard, which moves the decision to intercept silent API failures from the model to the Harness that runs tool calls, validating results with invariants that any correct API must satisfy and withholding violating values. The paper notes that APIs can fail silently, such as ignoring a misspelled filter parameter while returning HTTP 200 and a plausible but wrong result; existing approaches such as manifests, review models and LLM judges let the model find problems itself, but models often see them and still answer as usual. The study found silent failures in 11 of 15 public research APIs, with counts inflated by up to 691 times; under existing approaches more than 70% of answers were wrong, and for counts flagged by the review model the agent still reported 77.4%. On 7 open-source models, SilentGuard reduced wrong answers under the target failures from 74.0% to 1.7%, with no loss on failure-free tasks; on 3 frontier models it reduced them from 70.2% to 0.5%.