Skip to content

Where the Evidence Hides: Measuring Privacy and Diagnosability of Exported Agent Trajectories

Where the Evidence Lives: Measuring Privacy and Diagnosability of Exported AI-Agent Traces

arXiv: 其他 AI 安全论文· arXiv:2611.20389· Chang Xu, Hao Wang, Gedi Zhou, Junyuan TanDuke University; Stanford University; University of California, BerkeleyIndexed by this site · 2 days ago· Originally publishedEvaluation· Its place in the scan record →

Quick read

Evaluation and benchmarks

Based on the paper PDF (AI-generated); refer to the original

The authors build an executable benchmark to evaluate the privacy-diagnosability trade-off of AI agent execution-trace exports: pseudonymizing facts and execution domains preserves the fix, but row counts can still uniquely identify customers (97/240).

Questions
Sharing AI agent execution trajectories helps diagnose faults but may leak customer identity; conventional sanitization may both delete the evidence needed for repair and leave recoverable encoded identifiers.
Methods
The authors build an executable benchmark: three synthetic multi-tenant workflows, seven injected fault types, 15 export policies, repair tested with a rule program and an independent constraint program, and leakage tested with identifier matching, linkage, and row-count attacks.
Experiments and results
Comparison facts and exec-scoped pseudonyms preserve all 210 fault tasks for the rule program, while regex/schema succeed on only 140; however, under all policies the row-count attack uniquely identifies 97/240; the independent program loses 3 repairs with comparison facts.
Limitations and future work
The authors note that the evidence comes from three synthetic workflows, seven failure types, and bounded edits; pseudonymization covers only the declared syntax; the language model evidence uses only a small model. The experiments did not test stronger auxiliary data or model-based re-identification.
Experimental setupQwen2.5-3B-Instruct · 作者构建的三工作流合成基准(file、reconciliation、aggregation) · fault-task success、recovery et al.
Threat model
The attacker obtains the exported traces and public information, with the goal of identifying the requested customer or link from 32 candidate directories across two executions; the known-volume attacker additionally knows the row count of each candidate. The diagnostician only sees the export and the public operation list, with no private labels.
Evaluated models
Qwen2.5-3B-Instruct
Benchmarks
作者构建的三工作流合成基准(file、reconciliation、aggregation)
Metrics
fault-task successrecoveryunique identificationtarget exposureAUC
AI summaryResearchers propose an executable benchmark that measures how much customer identity agent trajectory exports leak and whether remediation schemes can restore correct outputs. Results on synthetic multi-tenant workflows show that conventional redaction may delete the evidence needed for remediation while leaving recoverable encoded identifiers. Two export schemes (matching on comparative fact reports, or using consistent pseudonyms within a single execution) both preserved the original diagnostic program's remediation, and the tested identifier matchers could not identify the requested customer from them; an independently written program produced correct outputs for 39 of 42 failure cases under comparative fact export, and for all of them under execution-scoped pseudonyms. However, if an attacker knows each customer's row count in advance, under each of 15 export policies they could uniquely identify the requested customer 97 times out of 240 executions; removing selected counters from the comparative fact export blocks this attack while preserving remediation. Engineers can use the benchmark to test diagnostic programs' remediation capability and the leakage risk under the same export.

Researchers propose an executable benchmark that measures how much customer identity agent trajectory exports leak and whether remediation schemes can restore correct outputs. Results on synthetic multi-tenant workflows show that conventional redaction may delete the evidence needed for remediation while leaving recoverable encoded identifiers. Two export schemes (matching on comparative fact reports, or using consistent pseudonyms within a single execution) both preserved the original diagnostic program's remediation, and the tested identifier matchers could not identify the requested customer from them; an independently written program produced correct outputs for 39 of 42 failure cases under comparative fact export, and for all of them under execution-scoped pseudonyms. However, if an attacker knows each customer's row count in advance, under each of 15 export policies they could uniquely identify the requested customer 97 times out of 240 executions; removing selected counters from the comparative fact export blocks this attack while preserving remediation. Engineers can use the benchmark to test diagnostic programs' remediation capability and the leakage risk under the same export.

Paper analysis

6 questions; open one to read the full answer

  1. What problem does this paper attempt to solve?

    When exporting AI agent trajectories for diagnosis, how to measure simultaneously what it leaks and what can still be repaired.

    Core question: when execution trajectories of shared AI agents are used for fault diagnosis, engineers need to know what the recipient can fix and what an attacker can infer from the same export.

    • Problem scenario: when an agent on shared data infrastructure returns another customer's record, the engineer must decide how much of the execution trajectory to share; attributing the error requires comparing the request with the record, but sharing both exposes identity (Introduction).
    • Shortcomings of existing methods: the authors state that deleting customer strings may delete the evidence needed for comparison while still leaving decodable encoded identifiers in notifications or tool parameters; prior work studied leakage, sanitization cost, or diagnostic telemetry separately, and did not measure repair outcomes and leakage on the same set of executions.
    • Core observation: even with a correct diagnosis, the proposed edit may leave the output wrong, so the paper re-runs the workflow after repair and scores the output, rather than looking only at the diagnosis conclusion.
    • Paper's contribution: an executable benchmark that jointly evaluates residual leakage and post-repair output of 15 export policies on synthetic multi-tenant workflows, and provides a reference-scored output endpoint.
    • Threat model:
      • Attacker goal: identify the requested customer among 32 candidate customer directories, or determine whether two trajectories share a customer/input unit.
      • Attacker knowledge: the export view and public information; the known-volume attacker additionally knows the row counts of each candidate; neither the diagnostic program nor the attacker has access to private target labels or attack-time secrets.
      • Victim system: three workflows using synthetic identities and faults, with real file/SQLite/CSV operations.
  2. What related research exists?

    Related work falls into four groups: trajectory anonymization, agent privacy and auditability, fault and task evaluation, and sanitization mechanisms.

    Trace anonymization

    • Packet privacy [9], Crypto-PAn's consistent keyed IP mapping [10], log anonymization [11], and unconventional identifier forms [12]—focus on the risks of trace publication.
    • Secret Sharer [13] tests model memorization, and trace-canary auditing [14] tests evidence of weight memorization; the authors note that the identification endpoint in this paper concerns a single execution and the requested customer in a directory.

    Agent privacy and auditability

    • LOPSIDED [15] pseudonymizes prompts and reconstructs responses; the authors note that this paper evaluates post-execution exports.
    • AgentLeak [1] tests internal leakage and redaction cost; RedAct [2] protects procedural knowledge and evaluates the usability of audit fields and human ratings.
    • AgentTelemetry [4] counts telemetry-guided patch interventions in a SWE-bench Lite case study; the authors note that this paper pairs replay outputs with leakage.

    Failure and task evaluation

    • AgentChaosBench [17] tests localization under runtime failures and redacted telemetry; TraceElephant [18] provides 220 human-checked public benchmark failure traces, and the authors note that it lacks the requested-customer labels of this paper.
    • TelemetrySuffBench [3] tests the sufficiency of paired views for failure tracing; the authors note that this paper combines diagnosis, post-decision output scoring, and leakage/linking.
    • AgentDojo [19] and computer-use probes [20] test injection safety; τ2-Bench [21] tests task utility; the authors note that the edits in this paper are bounded configurations.

    Redaction mechanisms and baselines

    • Source-aware redaction [16], the identifier detectors Presidio [6] and Scrubadub [7], and OpenTelemetry [8] provide removal mechanisms.
    • Baselines and benchmarks: the paper's comparisons are five post-hoc comparators including default Scrubadub, default Presidio, and dictionary substitution, evaluated on the same set of executions; there is no independent external benchmark.

    The authors position this paper as: conducting an executable evaluation of the residual leakage of these mechanisms together with diagnosis and reference scoring outputs on the same set of executions.

  3. How does the paper address this problem?

    The benchmark injects seven failure types across three synthetic workflows and simultaneously measures repair success and multi-channel leakage for 15 export policies.

    Overall approach: build an executable benchmark in which a trusted exporter prepares the same execution as multiple exports, a diagnostic program proposes bounded edits based only on the export, and a trusted executor replays them on a copy of the original workspace with scoring by a private reference output; the same set of exports is then given to multiple leakage attack endpoints.

    Roles, outputs, and evidence

    • The diagnostic program sees the export and a public list of allowed operations, and may select at most two configuration operations in one stage; if there is an edit, the executor re-runs the entire workflow, and if there is no edit, the original output is scored; the decision is sealed before the private reference output is scored.
    • Task success means the actual output passes the reference scorer; recovery means an originally failing workspace becomes correct; the two have different denominators because some injected faults do not affect the correctness of the original output.
    • Evidence falls into three categories: structural evidence (scan/write settings), value evidence (amount changes), and attribution evidence (whether a record belongs to the requested customer).

    Workloads and faults

    • The three author-built workflows use synthetic identities and faults, with real file, SQLite, and CSV operations: file (parse CSV, output JSONL, and validate a manifest), reconciliation (import rows, reconcile, export CSV), aggregation (read partitioned transactions, join dimension tables, aggregate decimal amounts, write a report).
    • Each workflow has seven fault families with two variants each: adapter settings, stale binding, incomplete scan, wrong write mode, value change, wrong customer, cross-customer contamination.
    • A unit = workflow × seed × input instance; each unit has one clean case plus seven faults; the eight cases share a target, a workspace, and 32 candidate directories, with only the target and two neighbors having rows and the rest being zero-capacity decoys; notifications represent identity as URL-query e-mail, base64 JSON headers, split fields, or at/dot obfuscated text.

    Export policies and diagnostic programs

    • 15 policies (Table II): raw, regex (five frozen patterns), schema (removing fields such as owner_id), typed projection, metadata-only and two ablations, comparison facts (optionally with run-scoped or exec-scoped owner tokens), dictionary substitution, and two encoding-aware pseudonym domains (canon-global, canon-exec).
    • Comparison facts report whether the owner at each stage matches the request (all/some/none/unknown/not applicable), distinct-owner counts, whether amounts are whole dollars, negative-amount counts, and stage pair edges; optional owner tokens use HMAC (split/run key or execution-handle key), and the attacker does not have the key.
    • The rule program (C1) checks attribution first, then metadata, then values; the independent constraint program (C4) enumerates allowed single-stage configurations, ranks them by fewer operations and fewer irrelevant changes, acts only on a unique optimal plan, and otherwise abstains.

    Leakage endpoints and protocol

    • Identifier matching (A0/A1) ranks the 32 candidates: basic matches raw, lowercase, URL, base64, hex forms and length buckets; normalized additionally applies Unicode compatibility normalization, etc.; unique identification requires the target to be solely highest-scoring, and direct exposure requires a score of at least 10.
    • Linkage attacks (A2/A2b) determine whether two trajectories share a customer and input unit, measured by AUC; known-volume matching (A3) compares rows_in with the candidate's known row count and succeeds only when the target uniquely matches.
    • The main study uses disjoint seeds: 576 development, 48 validation, 240 final executions (210 faulty + 30 clean), with 10 original policies producing 2,400 final views; the main key derives identities and is published after scoring.
  4. What experiments did the paper conduct?

    Repair and leakage of 15 exports measured on 240 executions in the main queue, plus an independent program, link/count attacks, and a language model check.

    Experimental setup

    • Subjects: 15 export policies (Table II); the diagnosticians are a rule program (C1) and an independent constraint program (C4); an exploratory model classifier (C3); the language model study uses Qwen2.5-3B-Instruct (4-bit, CPU, greedy).
    • Benchmark: three synthetic workflows × seven fault families × two variants; the main cohort has 240 final executions (210 faulty, 30 clean), and 10 original policies produce 2,400 views in total.
    • Metrics: fault-task success (/210), recovery (/190 initial failures), normalized unique identification and target exposure (/240), linkage AUC, known-volume unique match (/240).
    • Adjudication: outputs are adjudicated by a private reference scorer; leakage is adjudicated by a frozen matcher and attack rules; the independent program's source code and criteria were frozen before its 48-case test.
    • Controls: oracle applies the correct fix, no-op keeps the original output; five post-hoc comparators including Scrubadub and Presidio are run on the same set of executions.
    • Repetition and samples: the independent program uses a separate 48-case cohort; the language model study uses 96 development + 144 main executions, plus 144 post-hoc executions.

    Main results

    According to Table III (main cohort, all retaining full metadata; * denotes post-hoc same-case evaluation):

    Swipe horizontally to read the table

    ExportSuccess /210Recovery /190Unique ID /240Exposure /240
    Raw21019065240
    Regex140120225163
    Schema140120221226
    Scrubadub*210190153240
    Presidio*184164202235
    Metadata-only1159500
    Comparison facts21019000
    Exec-scoped pseudonyms*21019000
    • The authors state that comparison facts and the two pseudonym domains achieve 210/210 fault success and 190/190 recovery, while regex/schema achieve only 140/210 and 120/190.
    • The authors note that raw's unique identification is only 65/240 because the target and a neighbor tie for the highest score, but it exposes the target in every case; the zero values for pseudonyms and facts measure only the coverage of the tested syntax.
    • The authors note that all policies still allow known-volume unique matching in 97/240.

    Held-out test of the independent constraint program

    • What was tested: on a separate 48-case cohort, whether an independently written constraint program can repair using each export (Table IV).
    • Results: raw and exec-scoped pseudonyms achieve 42/42 fault success and 36/36 recovery; comparison facts 39/42 and 33/36; regex/schema 28/42 and 22/36; all five exports achieve 6/6 with no false positives on clean cases.
    • Authors' interpretation: all three fact losses are abstentions (two file-workflow delimiter faults lack localization evidence, and in one case a redundant amount constraint causes a plan tie); the authors state this does not prove comparison facts are inherently insufficient, but shows that exporters should test repair with the program that will use the export.

    Linkage and counting attacks

    • Linkage: exec-scoped pseudonyms and facts have the same identifier-aware AUC on file and reconciliation (0.8750, 0.5000), and on aggregation pseudonyms are 0.9633, higher than facts at 0.6585; run-scoped pseudonyms reach 1.0000 on all three workflows; the authors state they did not isolate the contribution of metadata summaries.
    • Generic linkage: generic linkage on facts with a run-scoped owner token gives AUCs of 0.7797, 0.7898, 0.7698, below the protocol-predicted lower bound of 0.95; the identifier-aware attack on the same export reaches 1.0000, 1.0000, 0.9971.
    • Counting ablation: after removing the seven count fields, the known-volume attack has no match in 240/240 views, and rule-program detection and fault success remain 210/210 with recovery 190/190; the authors state this result is specific to the stated capacity rules, and linkage and the independent program were not tested under this ablation.

    Language model check and other analyses

    • Language model study: Qwen2.5-3B-Instruct generates trajectories, with 912/967 steps calling validly (94%); fact detection is 70/72 structural, 17/18 value, 36/36 attribution faults with zero unique identification; regex/schema detect 0/36 attribution faults but uniquely identify 113/144 and 137/144; protocol criteria are all met.
    • Interactive content exposure: when only step index, explanation, tool, parameters and response are retained, schema sanitization still exposes the target in 127/144 main executions and 130/144 post-hoc executions; the authors state these retained simulated fields do not prove external delivery.
    • Exploratory model diagnosis: facts 30/144 correct vs metadata-only 26/144 (McNemar p = 0.289); the authors state facts show no demonstrable improvement.

    Other ablations and analyses

    • The metadata rule makes consistent decisions across eight full-metadata policies (0/240 disagreement), detecting 115/120 structural faults; 110/120 after removing relations, and 75/120 when measuring only.
    • The rule program detects all 60 wrong-customer/contamination faults under raw and facts, and 0 under regex, schema, and metadata-only.
    • Regex/schema detect 20/30 value faults; the rule program adds five amount-scale detections under raw, regex, schema, and facts.
    • All policies pass 30/30 clean cases (Presidio has two false positives); oracle and no-op pass 210/210 and 20/210 faults, respectively.
    • Dictionary substitution leaves 161/240 normalized unique matches, and its token-aware linkage extractor reaches AUC 1.0000 on all three workflows.
  5. What are some points that can be further explored?

    The authors list limitations including synthetic evidence and syntax coverage; the experiments cover three workflows, 15 policies, and a limited set of attack endpoints.

    Limitations and future directions noted by the authors

    • Evidence comes from three synthetic workflows, seven injected fault families, and bounded edits, and the public defaults narrow the repair search space (Section VI).
    • The rule program and facts share a selected vocabulary, and the independent diagnostic program tested only the first fault variant on a different cohort (Section VI).
    • Pseudonyms cover only the declared syntax searched by the matching attacks; stronger auxiliary data or model-based re-identification are outside these tests (Section VI).
    • The language model evidence uses only one small runbook-guided model and environmental faults (Section VI).
    • The main study and the model study retained protocols, checklists, and deviations, but their prospective protocol timing cannot be independently confirmed (Section VI).
    • Future directions: testing alternative count summaries and execution-keyed summaries could extend these results (Section VI); the authors recommend that exporters separately audit encoded identifiers, counts, and linkage, and check post-repair outputs with the intended program and an independent implementation (Section VI).

    Scope of experimental coverage

    • There are 15 export policies tested, including five post-hoc comparators such as Scrubadub, Presidio, and dictionary substitution (Table II, Section III-D).
    • The main cohort is 240 final executions (210 faulty, 30 clean), the independent program uses a separate 48-case cohort, and the language model study is 144 main executions plus 144 post-hoc executions (Section III-D, V).
    • Leakage evaluation includes three types of endpoints: basic/normalized identifier matching, generic/identifier-aware linkage, and known-volume matching (Table II).
    • The language model study tests only fault classification and interactive exposure, not model-based repair (Section V).
    • The counting ablation did not test identifier-aware linkage, nor leakage on the independent program cohort (Section IV-C).
  6. Summarize the main content of the paper

    An executable benchmark shows: exports that preserve the fix can still leak customer identity, and row-count metadata is the remaining leakage channel.

    An evaluation/benchmark paper on the trade-off between privacy and diagnosability in exporting AI agent execution trajectories, which builds an executable benchmark: three synthetic multi-tenant workflows, seven injected fault types, 15 export policies, repair tested with a rule program and an independent constraint program, leakage tested with identifier matching, linkage, and row-count attacks, and outputs adjudicated by a private reference scorer.

    • Problem addressed: conventional sanitization may delete the attribution-comparison evidence needed for repair while leaving decodable encoded identifiers; engineers need to know what the recipient can fix and what an attacker can infer.
    • Method highlights: comparison facts delegate attribution comparison to the exporter, and exec-scoped pseudonyms let the diagnostician compare on its own; diagnostic decisions are sealed before scoring, and edits are replayed on a copy of the original workspace.
    • Main results: on the main cohort, comparison facts and exec-scoped pseudonyms preserve all 210/210 fault tasks and 190/190 recoveries for the rule program, while regex/schema achieve only 140/210 and 120/190 (Table III).
    • Independent program: the constraint program achieves 42/42 success under exec-scoped pseudonyms and 39/42 under comparison facts, with all three losses being abstentions; the authors state this shows exporters should test repair with the program that will use the export (Table IV).
    • Residual leakage: under all 15 policies, an attacker who knows row counts uniquely identifies the requested customer in 97/240; after removing the seven count fields, the attack has no match in 240/240 views and repair is unchanged; exec-scoped pseudonyms have linkage AUC 0.9633 on the aggregation workflow, higher than facts at 0.6585.
    • Authors' conclusion: an export may simultaneously remove identifiers, lose repair evidence, and still leak the requested customer; the authors recommend that exporters separately audit encoded identifiers, counts, and linkage, and check post-repair outputs with the intended program and an independent implementation.
Read originalarxiv.org