Skip to content

Auditable Authorization for LLM Agents: Empirical Evidence Across Model, Harness, and Workflow Layers

Auditable Authorization for LLM Agents: Empirical Evidence Across Model, Harness, and Workflow Layers

arXiv: 其他 AI 安全论文· arXiv:2611.19237· Bangjie Deng, Xiaoxuan Lu, Wenjun Chen, Chengge Wen, Yulin Wang, Jiayue Meng, Yao Yu, Liying Liang, Weijia Han, Ziwei Hong, Zidong YuUniversity of Pennsylvania; Columbia University; Northeastern University; University of Chicago; University of Rochester; Shanghai Jiao Tong University; University of Washington; Lehigh University; Syracuse UniversityIndexed by this site · 2 days ago· Originally publishedDefenses· Its place in the scan record →

Translation in progress — untranslated text is omitted.

Quick read

Evaluation and benchmarks

Based on the paper PDF (AI-generated); refer to the original

AI summaryThe researchers propose an auditable authorization method for LLM agents, formalizing human approval as refusal safety and bounded authorization integrity, and auditing at three execution boundaries: model selection invocation, Harness scheduling recovery, and workflow routing to protected effects. The experiments cover three models, five prompt sets, four runtime approval designs, two framework replays with CrewAI and n8n, and 100 public workflow sources. Results show that in the paired model panel, 7 of 300 approvals were not completed and the refused effects were zero; parallel confirmation lost approved branches in three tasks for each model; the recovery process missed 336/672 CrewAI approvals; and 4 of 59 comparable n8n workflows still executed after Reject. The authors state that comparing intent, generated invocation, dispatch, and native effects can locate the first observable divergence and its repair target, and that the resulting audit contract makes refusal safety and authorization integrity independently verifiable, turning human approval from a procedural formality into an executable control boundary between intent and real effects.

Questions
In critical agent systems, human approval is often the last means of control, but the interface recording a decision does not mean the decision controlled the work: approved work may not be completed, rejected effects may still occur, and real execution traces are difficult to make public and reproduce.
Methods
The authors formalize two obligations, refusal safety and bounded authorization completeness, and use paired Approve/Reject decisions, exact traces and native business-state witnesses (rather than LLM judges) to audit three models, four runtime approval designs and 100 public n8n workflows separately at the model, harness and workflow layers.
Experiments and results
In the paired panel, 7 of 300 approvals were incomplete and rejections leaked zero; Google ADK merged parallel approval lost one approved branch on 3/3 tasks; CrewAI recovery missed 336/672 approved effects; 4/59 n8n workflows still executed the protected effect after Reject.
Limitations and future work
The authors note: risk-oriented checkpoints and repeated transition episodes do not estimate incidence rates in deployment; the descriptive Wilson intervals are not a claim of independence; AI annotation agreement is low (κ 0.190–0.477), and native effects remain the primary endpoint; synthetic triage accuracy describes only its pre-declared fixed examples.

Related papers

Show 3 more papers
Read originalarxiv.org