LeakageBench: Document-level leakage risk in redaction of personally identifiable information in document images

PII redaction on document images is unsafe if even one in-scope identifier is missed on a page; entity-level F1 cannot substitute for this page-level risk.

• Scenario: The authors state that institutions often redact PII from scanned and rendered business documents. OCR errors, complex layouts, repeated identifiers, and visual noise make it different from annotation on clean text.
• Existing gaps: Most PII benchmarks measure precision, recall, and F1 on text spans; document understanding benchmarks usually target only selected key fields. The authors state that neither directly measures publication safety.
• Observation: The authors view false positives and misses as an asymmetric trade-off: false positives obscure useful content, while a single miss can make a page unsuitable for publication. Page-level leakage is a recall-oriented supplement, not a replacement for localization and typing F1.
• Proposal: LeakageBench is a challenge set of 500 pages with 11,954 GDPR-aligned annotations, distinguishing direct identifiers, linkage keys, and contextual re-identification surfaces, and uses a unified image-space interface to compare off-the-shelf OCR pipelines and OCR-free models.
• Task boundary: The evaluation target is visible PII on single-page document images. A system must localize each in-scope region and give its type, with optional text extraction; inference is per-page independent, matching is within a page, and no cross-page context is provided. The safety decision is whether any in-scope annotation remains unmatched.

The authors organize existing evaluations into four categories and argue that they do not directly address page-level release safety for document images.

Text PII detection and anonymization

• Span-level benchmarks: Pilán et al. (2022)'s TAB, Gambarelli et al. (2023)'s SPEDAC, Mo et al. (2025)'s DataSIR, Holmes et al. (2024)'s educational data PII competition, and Stubbs et al. (2015)'s i2b2 clinical narrative de-identification establish span-level detection and anonymization. The authors state that these works do not include document image artifacts, spatial localization, or page-level release safety.

Document image understanding

• Field extraction: FUNSD (Jaume et al., 2019), SROIE (Huang et al., 2021), CORD (Park et al., 2019), and DocVQA (Mathew et al., 2020) evaluate form understanding, receipt extraction, OCR, and visual question answering. BuDDIE (Wang et al., 2025) covers multiple commercial document tasks and includes person and address fields. The authors state that these benchmarks target task-specific extraction rather than exhaustive PII localization and residual redaction leakage.

OCR-related and OCR-free models

• Layout models: LayoutLM (Xu et al., 2019) and LayoutLMv3 (Huang et al., 2022) combine recognized text with two-dimensional layout; Donut (Kim et al., 2022) directly generates structured output from document images. The authors state that for redaction, the former propagates OCR and text-to-box alignment errors, while the latter must simultaneously learn visual reading and spatial localization.

Leakage-oriented privacy evaluation

• Residual exposure: PRvL (Garza et al., 2025) evaluates LLM-based PII redaction and residual exposure; ProPILE (Kim et al., 2023) studies privacy leakage with targeted probing; RedactBuster (Beltrame et al., 2024) discusses that redacted documents may still be vulnerable to downstream inference. The authors state that these works advance exposure-oriented evaluation but do not provide a localized document image benchmark with page-level release guidelines.

Baselines and data

• Comparison in this paper: Experimental baselines include Presidio (Tesseract), GLiNER variants on shared Textract, Amazon Comprehend PII, OpenAI Privacy Filter, Google DLP, and Qwen3-VL-32B, InternVL3-38B, GPT-5.4, GPT-5.5 and their Code Interpreter configurations. The evaluation set is LeakageBench, with pages sampled from OCR-IDL, VRDU Ad-buy Forms, and FCC public documents.

The authors place this work under a unified image-space interface, evaluating both types of systems simultaneously, and put entity-level F1 and page-level leakage in the same redaction task.

The overall approach is to make redaction a high-recall localization on the image, rather than only labeling on OCR text. LeakageBench compares off-the-shelf systems with a unified prediction format, and uses DocLeak to judge whether a page is still unsuitable for release.

Task and matching

• Input: a single page from a scan, screenshot, or PDF render. The system marks each visible region containing PII, assigns a type, and may optionally extract the original text. gold contains image boxes, fine-grained types, risk groups, and optional verbatim values.
• Prediction: axis-aligned boxes, PII type, optional value, optional confidence (0 to 1). Boxes from OCR pipelines come from word boxes; OCR-free models predict directly from the image.
• Matching: one-to-one within a page. A pair can be matched only if IoU is at least τ, selected greedily from high to low IoU, and each prediction and gold is used at most once. Localization matching ignores type; typed matching additionally requires matching types. When values are present, values are evaluated only on already matched pairs.
• Heading criteria: DocLeak in Tables 3–4 uses type-aware matching: gold counts as protected only if IoU qualifies and the type is the same. Diagnostics additionally use type-agnostic matching, because a box with the wrong type but correct location still covers the identifier.

Metrics

Page-level leakage is defined as

DocLeak=(1/|D|)∑_{d∈ D}1[d leaks]

D is the set of pages containing at least one in-scope identifier; a page leaks if any in-scope gold is unmatched. DocLeakall covers all PII, DocLeakcrit covers only Direct and Linkage. This metric does not penalize extra predictions, so it is reported together with F1: F1loc is type-agnostic page-macro-average localization F1; F1type additionally requires correct type and is macro-averaged over gold types. Unless otherwise stated, τ=0.75. One-to-one matching prevents one box from scoring for multiple golds.

Data and schema

• Sources: 500 pages, 291 public commercial documents, which the authors position as a challenge set rather than representative of all deployment scenarios. Source A is 103 documents, 113 pages from OCR-IDL; Source B is 107 documents, 222 pages from VRDU Ad-buy Forms; Source C is 81 documents, 165 pages from FCC free-form files.
• Scale: 11954 PII instances, averaging 23.90 per page; Direct+Linkage totals 4315, averaging 8.63 per page (Table 2). Other is 239, such as signatures and agency codes.
• Groups: Direct includes names, addresses, emails, phones; Linkage includes lookup keys such as various numbers; Contextual includes dates, locations, organization names, Business ID (Table 1). The authors state that Contextual alone is not necessarily personal data, but is a surface that amplifies re-identification when combined with direct identifiers or linkage keys.
• Diagnostic views: Figure 1 is a histogram plus CDF of the number of PII per page. Figure 2 divides each page into non-overlapping 14×14 pixel blocks with stride 14, yielding 60×60 = 3600 blocks, aggregating all PII and critical PII separately. Page type totals are invoice 229, form 39, email 90, letter/correspondence 62, document page 47, other 33 (Table 7).

Annotation and quality control

• Process: the first two authors randomly split the 500 pages equally and annotated independently. Boxes tightly enclose visible text, values are transcribed verbatim without normalization; multi-line uses the line-break notation specified in the paper.
• Agreement: each person cross-annotated 10% of their assigned pages, 50 pages in total; disagreements were jointly adjudicated into a single gold. Alignment requires IoU≥0.75 and the same fine-grained type, otherwise it counts as disagreement. On this subset, Cohen's κ=0.979.

Evaluation protocol

• Off-the-shelf systems: no training or tuning on LeakageBench. In full-schema scoring, unsupported types, invalid outputs, and out-of-bounds boxes count as misses; unparseable pages count as empty predictions. Appendix B records OCR engines, detector settings, schema mappings, and conversion rules.
• OCR-related: Presidio uses Tesseract. The other stronger detectors, except Google DLP, share Amazon Textract word boxes, including constrained GLiNER's Presidio-best, GLiNER-base, GLiNER multi-PII, NVIDIA GLiNER PII, GLiNER2, Amazon Comprehend PII, and OpenAI Privacy Filter. Detection is done on OCR text, then mapped back to the minimum bounding rectangle of the matched word boxes; misaligned detections are discarded.
• OCR-free: Qwen3-VL-32B, InternVL3-38B, GPT-5.4 single-pass vision, and GPT-5.5's single-pass vision and Code Interpreter configurations. The prompt gives the full schema, and coverage is 1.0 by configuration, which does not mean detection is reliable. Boxes are first normalized to 0–999 coordinates, then converted to pixels. GPT-5.5 uses high image detail and medium reasoning effort; the tool configuration must inspect the page and may crop or zoom before the final prediction. Appendix C gives the full prompts.
• Trained models not included: the authors leave LayoutLM-style document NER to a subsequent trained-baseline track, because they typically require supervised fine-tuning, task heads, OCR token alignment, and training splits.

Experimental Setup

• Systems: The 14 off-the-shelf configurations from Table 3, none trained on this set. OCR-dependent are Presidio (Tesseract), and Presidio-best, GLiNER-base, GLiNER-multi-PII, NVIDIA GLiNER PII, GLiNER2, Amazon Comprehend PII, OpenAI Privacy Filter sharing Textract; Google DLP uses vendor OCR. OCR-free are Qwen3-VL-32B, InternVL3-38B, GPT-5.4, GPT-5.5, GPT-5.5+Code Interpreter.
• Data: LeakageBench, 500 pages, 11,954 annotations, 4,315 critical identifiers. Source composition in Table 2.
• Metrics: F1loc, F1type, DocLeakall, DocLeakcrit; Valid is the proportion of pages that are parseable and schema-compliant. Main results at τ=0.75, DocLeak is type-aware. Unsupported types and invalid outputs count as missed detections.
• Matching: One-to-one matching with gold boxes via IoU; the paper does not use a separate LLM judge, nor does it report repetitions or query budgets.
• Annotation quality control: 50 pages double-annotated, κ=0.979.

Main Results

The table below is excerpted from Table 3 (full schema, τ=0.75). Omitted are Presidio-best, GLiNER-base, GLiNER-multi-PII, GLiNER2, OpenAI Privacy Filter, and Qwen3-VL-32B. Among them, GLiNER-multi-PII has F1type 0.091 and DocLeakcrit 0.972; Qwen3-VL-32B has Valid 0.666.

Swipe horizontally to read the table

System	F1loc	F1type	DocLeakall	DocLeakcrit	Valid
Amazon Comprehend PII	0.304	0.085	0.992	0.968	1.000
NVIDIA GLiNER PII	0.288	0.109	0.992	0.986	1.000
GPT-5.5 + Code Interpreter	0.249	0.119	0.990	0.968	1.000
Presidio	0.137	0.048	0.994	0.990	0.996
GPT-5.5	0.090	0.050	1.000	0.990	1.000
Google DLP	0.076	0.013	1.000	1.000	1.000
GPT-5.4	0.044	0.031	1.000	1.000	1.000
InternVL3-38B	0.012	0.019	1.000	1.000	0.540

• The authors state that stronger detection and tool assistance improved localization but did not make most pages publishable. The authors report a type-aware DocLeakcrit range of 0.968–1.000.
• When any IoU-qualifying box is counted as protected, the range becomes 0.945–1.000. The authors argue that high leakage is not mainly due to type mislabeling.
• The authors state that leakage is not marginal: leaking pages still retain on average 5.3–8.7 critical identifiers per system; the system with the lowest unmatched count misses on average 5.3 on leaking pages (median 4, P90 11), which the authors say is approximately 66% of applicable page critical identifiers. Table 13's mean missed is 5.308–8.652, mean proportion is 0.656–0.990.

Leakage After Supporting Types

• What was tested: Table 4 retains only gold types each system can output. Restricted coverage is 0.208 (Presidio, Presidio-best, Amazon Comprehend PII) or 0.250 (Google DLP, OpenAI Privacy Filter); GLiNER2 is 0.958. The authors do not rank leakage on restricted blocks because the supported type sets differ.
• Results: For the three GLiNER systems with full schema and shared Textract, Direct page leakage is 0.942–0.954, Linkage is 0.967–0.973. GPT-5.5+Code Interpreter's Direct/Linkage/Contextual is 0.887/0.964/0.908.
• Authors' interpretation: The authors state that failures are not just taxonomy mismatches; OCR errors, duplicate identifiers, span fragmentation, and box reconstruction also contribute to residual exposure. The authors state that even with label sets available, sparse lookup keys, dense tables, OCR fragmentation, and layout variation remain difficult.

Failure Attribution

• OCR-related: Under type-agnostic matching, leaked critical identifiers are attributed to OCR misses, detector misses with zero overlap, or boxes/alignment failures with overlap but IoU below 0.75. One-to-one conflicts are at most 0.2%. Seven of nine systems are dominated by boxes/alignment, accounting for 45.9%–84.5% of their attributable leakage; detector misses account for 15.5%–48.8% (Table 14). GLiNER-multi-PII has detector misses 48.8% and boxes/alignment 34.9%; OpenAI Privacy Filter has 44.4% and 36.6%. Google DLP's OCR miss share is not reported because vendor OCR text is unavailable.
• OCR recovery: Of 4,280 critical annotations with original text, Textract fails to recover 623 (14.6%), Tesseract fails to recover 765 (17.9%). Of 4,315, 35 have no verbatim gold (Table 15). The authors state that differences among shared-Textract pipelines mainly arise after OCR.
• OCR-free: In Table 5, interface failures account for 27.8% of Qwen3-VL-32B's leaked critical instances, 36.9% for InternVL3-38B, and 0 for GPT configurations. Of remaining leakage, sub-threshold boxes account for 51.7% for Qwen, 83.5% for GPT-5.4, 96.1% for GPT-5.5, and 98.9% for the tool-assisted configuration. The authors state these percentages are composition of remaining errors, not overall localization failure rates. Tool assistance reduces Direct leakage from 0.956 to 0.887 and Contextual from 0.986 to 0.908, while Linkage remains 0.964. The authors use Figure 3 to illustrate transcription, detection, localization failures, and one tool-assisted correction; the legend shows green gold boxes, orange dashed sub-threshold boxes, blue tool-assisted boxes, and magenta dotted OCR regions.

IoU Sweep

• What was tested: Appendix G repeats full-schema scoring at τ of 0.50, 0.75, 0.95 (Table 10).
• Results: According to Table 10, the lowest DocLeakcrit is 0.775, corresponding to GPT-5.5+Code Interpreter at τ=0.50, with F1loc 0.614. At the same threshold, Amazon Comprehend PII's DocLeakcrit is 0.901. At τ=0.95, all systems in the table have DocLeakcrit 1.000 except Amazon Comprehend PII at 0.998.
• Authors' interpretation: The authors state that entity-level F1 varies greatly with τ, but document-level leakage remains high across thresholds.

Other Ablations and Analyses

• OpenAI Privacy Filter's Valid is 0.806 (Table 3); Linkage type coverage is 0.077 (Table 8), and Linkage leakage on supported types is 0.994 (Table 4).
• Table 9 full schema: Google DLP's Direct and Linkage are both 1.000; Amazon Comprehend PII's Linkage is 1.000 and Direct is 0.908.
• Table 13: leaking pages' mean missed is 5.308–8.652; applicable pages' mean proportion is 0.656–0.990.
• Table 4: GLiNER2's type coverage is 0.958, with the proportion of pages containing unsupported types at 0.094.
• Table 8: Presidio's Direct coverage is 1.000, Linkage coverage is 0.000, and Contextual coverage is 0.333.
• Double-annotated 50 pages, κ=0.979 (Sec 4.6).

Limitations and future directions noted by the authors

• Scope: The authors state that this is a 500-page challenge set from public commercial documents, not a representative sample of all redaction scenarios; it does not cover heavy handwriting, medical, legal discovery, multilingual, or non-US documents (Limitations).
• Within-page evaluation: Evaluation is per-page and does not measure multi-page context inference; Sec 3.1 states that evaluating multi-page context inference remains future work. Sec 5 leaves layout-aware models requiring fine-tuning to the trained-baseline track.
• Policy dependence: Direct and Linkage are the primary release safety targets; whether to redact Contextual depends on applicable policy (Limitations).
• Matching criteria: The headline DocLeak uses type-aware matching and IoU τ=0.75; boxes with a type error but correct localization count as leakage. The authors also report that under type-agnostic matching, DocLeakcrit remains between 0.945 and 1.000 (Limitations).
• Geometry and utility: Axis-aligned boxes and IoU do not directly measure pixel coverage of slanted, handwritten, or irregular regions. F1 penalizes spurious predictions, but the authors state that document-level over-redaction or utility metrics are not defined (Limitations).
• Attribution and configuration: 35 key annotations lack available source text, there are no span-level alignment logs, Google DLP's vendor OCR is not saved, and box errors and alignment errors cannot be fully separated. OCR-free results correspond to specific models and inference configurations, whose API behavior may change (Limitations).

Experimental coverage

• Evaluation covers 14 off-the-shelf configurations in Table 3, none trained or tuned on LeakageBench (Sec 5–6).
• Data consists of 500 pages, 291 documents, from three sources; 11,954 annotations, of which Direct 3,779, Linkage 536, Contextual 7,400, Other 239 (Table 2).
• Main metrics at IoU τ=0.75; Appendix G additionally reports τ of 0.50, 0.75, 0.95 (Table 10).
• Cross-annotation covers 50 pages, with agreement defined as IoU≥0.75 and fine-grained type agreement, κ=0.979 (Sec 4.6).
• The paper does not report value accuracy aggregates, per-system query counts, or numbers of repeated experiments. Sec 3.3 specifies that when a value exists, value is only evaluated on matched pairs.

LeakageBench uses page-level leakage to measure whether document image PII redaction is releasable, rather than only comparing entity F1.

• Problem: Redaction on scans, screenshots, and PDF renders is affected by OCR, layout, and visual noise. The authors state that missing one in-scope identifier makes the page unsafe; false positives and false negatives are asymmetric.
• Data and protocol: 500 pages, 11,954 annotations, divided into Direct, Linkage, Contextual. Predictions must be image boxes. Within a page, one-to-one matching at IoU 0.75; DocLeak checks whether any unmatched gold remains, and F1loc and F1type jointly constrain spurious predictions and types. None of the systems are trained on this set.
• Main results: In Table 3, the highest F1loc is 0.304 for Amazon Comprehend PII, the highest F1type is 0.119 for GPT-5.5+Code Interpreter, and both have DocLeakcrit of 0.968. Tool assistance raises GPT-5.5's F1loc from 0.090 to 0.249, while Linkage leakage remains 0.964. Under type-agnostic matching, DocLeakcrit remains 0.945–1.000.
• Failure sources: The authors attribute failures to schema coverage, OCR recoverability, detection, interface effectiveness, and spatial localization. Textract's unrecoverability rate for 4,280 key annotations with values is 14.6%. Full-coverage GLiNER's Linkage page leakage remains 0.967–0.973 (Table 4). The lowest DocLeakcrit in Table 10 is 0.775, occurring at τ=0.50 for tool-assisted GPT-5.5.
• Authors' conclusion: The authors believe this set is better suited as a redaction safety challenge set rather than a leaderboard with small score differences. The authors state that these results point to systems that simultaneously have broad schema coverage, robust visual reading, and tight high-recall localization. They plan to release images and annotations for research purposes under a Data Use Agreement, and state that the schema does not define legal compliance.