Optimizing agent planning to balance safety and autonomy

Researchers propose a safety-aware agent design and an autonomy metric, so that deterministic system-level defenses maintain safety while reducing reliance on human oversight. The paper notes that indirect prompt injection threatens agents executing critical actions; such defenses can provably block unsafe actions by enforcing confidentiality and integrity policies, but currently reduce task completion rates and increase token usage compared with probabilistic defenses, and their benefit of reducing human oversight is ignored by existing evaluations. The autonomy metric measures the proportion of critical actions an agent can execute without human-in-the-loop (HITL) approval while maintaining safety. The new design introduces richer human approval interactions and explicitly plans task progress and policy compliance, implemented on top of existing information flow control defenses; experiments on the AgentDojo and WASP benchmarks show it can improve autonomy without sacrificing task completion rates.