LLM-Attacker: A Unified Red-Teaming Framework for Privacy Evaluation of Split and Distributed LLM Systems

1. 01What problem does this paper attempt to solve?

   Privacy evaluation of split LLMs is hard to compare; LLM-Attacker records observation, knowledge, and scoring rules as verifiable experiments.

   Privacy evaluation of split language models lacks a checkable experimental contract: when the messages the attacker sees, the weights they hold, the corpus, and the recovery metric differ, conclusions are hard to compare.

   • Scenario: split learning keeps private text on the client and sends each token's activation to the server; during training the server returns gradient. The authors state that both types of messages contain private text information, and existing attacks can reconstruct text from gradient, activation, or both.
   • Existing shortcomings: the authors state that results depend on which messages the attacker sees, which weights they know, how much public text and computation they use, and the dataset and recovery metric. A defense may appear effective when the attack would have failed anyway; average token scores may mask exact recovery of a whole document or a sensitive span.
   • Question to answer: what must an evaluation record and test so that another researcher can verify the conclusion, compare fairly, and extend it.
   • What this paper provides: LLM-Attacker records the attacker's observations, knowledge, and resources; compatible attacks replay the same batch of messages, and predictions are saved for independent scoring. Scoring requires an associated no-defense reference first, and the report also marks untested attacks. The authors state that the GPT-2 experiments here are shared with a companion study, and are used to see how the evaluation responds to changes in observation.
   • Threat model: the main attacker passively observes one split point.
     • Goal: reconstruct private text from the messages crossing the split point.
     • Knowledge: by default holds a public copy of the client layers, i.e. the downloadable pretrained layers; a stronger condition instead uses the trained copy, including the client's fine-tuned embedding, and additionally provides public calibration text.
     • Capabilities: sees activation at training and inference, and the gradient returned during training; does not modify traffic, and does not receive private text, next-token labels, or the server layers. The capture harness uses labels to compute loss, but does not give the labels to the predictor.
     • Difference from stronger roles: this observation contract differs from an active server that controls the returned tensors, and from a server that directly holds the loss labels. The labelled-example, key-solving, active-server, and side-channel modules separately declare additional privileges.

   Collapse this answer

2. 02What related research exists?

   The authors position LLM-Attacker as an experimental contract for split systems, not as a reproduction of all cited protocols.

   In Related Work, the paper divides related work into evaluation frameworks, reconstruction attacks, and earlier split learning attacks.

   Evaluation frameworks

   • VFLAIR-LLM (Gu et al., 2025): puts configurable split inference, fine-tuning, and attacks and defenses in the same framework.
   • garak (Derczynski et al., 2024): organizes safety probing of large language models with generator, probe, detector, and report.
   • ML Privacy Meter (Murakonda and Shokri, 2020): measures training data privacy risk with membership inference.

   Text reconstruction attacks

   • Gradient reconstruction: DLG (Zhu et al., 2019), TAG (Deng et al., 2021), LAMP (Balunović et al., 2022) recover text from gradient.
   • Activation inversion: Vec2Text (Morris et al., 2023) and SipIt (Nikolaou et al., 2026) invert input from activation.
   • BiSR (Chen et al., 2024): a learned inverter then refined via gradient and activation, aimed at split fine-tuning. The authors compare the port and the archived reference implementation under the local single-seam contract, and compare the adapted version with other attacks at fixed budget.

   Early split learning attacks

   • Pasquini et al. (2021) and Erdoğan et al. (2022) demonstrated inversion, model stealing, and label inference.
   • Dong et al. (2025) discuss why depth itself does not guarantee privacy; Gu et al. (2026) discuss recovery in split inference. The authors state that these works motivate evaluating across different split depths and observation surfaces.

   Baselines and data

   • Baseline methods: activation-only, shuffled-gradient, random-candidate, backward-only heuristic, SipIt, and the most common token in calibration text; the BiSR comparison uses the archived reference implementation and refined BiSR.
   • Benchmarks/datasets: fixed 32-token WikiText-2, new held-out documents from the WikiText-103 training split, repository Python source code, templated synthetic clinical notes, pilot WikiText-2, and 32-token dialogue prefixes.

   The authors position LLM-Attacker as an experimental contract for split and distributed systems: explicit observation and knowledge, shared capture, associated controls, execution coverage, and versioned evidence. The remaining modules are written as local implementations or adaptations, with evidence levels recorded separately; the authors state this is not a reproduction of the complete protocol of every cited paper.

   Collapse this answer

3. 03How does the paper address this problem?

   Using logging, replay, and scoring with a no-defense reference, a privacy evaluation is made into a verifiable experiment.

   LLM-Attacker writes a privacy evaluation as an experiment that can be recorded, replayed, and independently scored, rather than reporting only a recovery rate.

   Components to record in an evaluation

   An evaluation is written as E=(T,V,O,K,B,C,M,G). T is the target model, split, defenses, and checkpoint; V is the attacker's position and privileges; O is the observation; K is the knowledge; B is the calibration, optimization, and search budget; C is the corpus, splits, and tokenization; M is the measurement results; G is the controls, uncertainty computation, and decision rules. These components are recorded in a fixed protocol, run configuration, and result records.

   Reading gradient changes O, obtaining the trained layer changes K, and swapping Wikipedia for code changes C. Attack comparisons keep the remaining components fixed and report each attack's budget. joint and activation-only also align the number of candidates per position and the beam width, so as to isolate the information in gradient by comparison. The authors note that aligning the search does not align total computation, because joint also uses gradient to rank candidates.

   Capture, replay, and module contracts

   Capture separates target execution from attack execution (Figure 1). split capture saves forward messages, and during training also saves the returned gradient, as well as session and order metadata. Compatible attacks replay these observations against the same checkpoint and documents. Other observation surfaces such as timing or model outputs use their own observation provider. Regenerating predictions and re-scoring saved predictions are two different checks.

   Each module declares the modes it supports, input requirements, threat metadata, and execution function; the registry rejects duplicate names. New attacks can enter the frozen comparison under the same input-output contract, and bring their own controls and budget. If a run is interrupted, completed measurements are still retained. The supplementary material writes the observation points as S1–S9, and after review adds S10–S14 (Figure S2, Table S1). By default the attacker only reads the activation and gradient at the split point; modules that read other surfaces assume a stronger attacker.

   Scoring and pass rules

   A simple guess for token reconstruction is the most common token in the attacker's calibration text. For applicable attacks with valid measurements, the test envelope takes the maximum of each attack's excess:

   L_run(E)=max_a100[U₉₅(p̂_a,n_a)−p_0,a(C)]

   After multiplying the bracket by 100, the unit is percentage points. U95 is a versioned one-sided Wilson upper bound, z=2.5392. This multiplier comes from a five percent error budget of nine known-plaintext probes, and is fixed for use with other attacks. The authors state it is not a simultaneous confidence guarantee for all 64 modules; token correlation also limits reading it as an independent binomial distribution. The authors state this gate is an operational screening convention, not a distribution-free privacy guarantee.

   • Defense passes: maximum excess at most 1 percentage point; at least three random seeds; the associated no-defense reference qualifies under scorecard-policy.v3. The reference must cover the same attacker family, and have one result with excess above 1 percentage point.
   • Insufficient evidence: a missing or unqualified control turns a candidate pass into INSUFFICIENT_EVIDENCE. A measured high recovery remains visible regardless of whether a control is available. When there is no valid attack, the envelope is unmeasured. The rules are fixed before defense testing.
   • floor: under finite samples, even when the point estimate is exactly at baseline, the upper bound still makes the excess positive. With small samples the floor can exceed 1 percentage point, and a failed gate must be read together with the estimate and interval. When the protocol and existing measurements are fixed, adding attacks can only raise the envelope or leave it unchanged.
   • No automatic generalization: whole-document and span recovery keep their own metrics. membership, timing, and agentic results do not pass through this token bound. A verdict must state the corpus, attacker set, and setting.

   Reconstruction methods in the main comparison

   activation-only fits a ridge decoder from public activation to token embedding, with 32 candidates per position, beam width 2, and recomputes activation to match. joint keeps 16 activation candidates, then adds 16 more from the negated gradient at the previous position; overlaps are filled from the activation list; the first position uses only activation candidates. The number of candidates and the beam width are aligned.

   shuffled-gradient shuffles gradient across documents; random-candidate replaces gradient-side candidates with random tokens. backward-only reads only gradient. SipIt searches by position; the authors state they implemented the published algorithm themselves, and that when the budget is exhausted it returns already-tried tokens, so the exact recovery guarantee no longer holds. Appendix S2 gives a four-step description.

   The white-box path knows the defense and optimizes a continuous embedding so that, after passing through the client layers and the real compressor, it matches the observation. inversion uses a linear mapping of the same kind, without beam search. BiSR is adapted to a single split: the inverter guesses first, then matches gradient and activation separately. The authors state that what is compared is each one's fixed budget, not an equal-compute ranking, and not the original two-cut protocol.

   Coverage, qualification, and evidence

   The three outputs answer different questions: the scorecard evaluates measured results under declared rules; the coverage report marks applicable threats as covered, partial, unmeasured, or out of scope; the module-validation matrix marks qualification as reference conformance, adaptation, real-model qualification, synthetic qualification, or unmeasured capability. An execution failure is not a negative attack result; registration alone is also not real-model validation. The full matrix is Supplementary Table S15, with 64 modules in total, including three orchestration helper modules.

   The evidence process checks files against retained hashes, recomputes metrics from predictions, and regenerates predictions from fixed inputs. The authors state that a hash chain without signatures can be recomputed, and therefore cannot authenticate an untrusted producer; scientific validity depends on the protocol, controls, and reproducible analysis. A change to the attack set or scoring policy yields a new version.

   Collapse this answer

4. 04What experiments did the paper conduct?

   On GPT-2 layer 6 public layer, after adding gradient, token goes from 94.20% to 97.38%, and whole-document goes from 13.71% to 37.77%.

   The main experiment on controlled GPT-2 changes whether the attacker sees gradient, to see how the recorded evaluation changes.

   Experimental setup

   • Model under test: the main reconstruction is GPT-2 124M, split at layer 3 or layer 6. The qualification procedure in Section 6 also writes Qwen3-0.6B, Qwen3-8B, Qwen3.5-27B, Qwen3.5-35B-A3B, Mistral-7B-Instruct, Mistral-NeMo-12B-Instruct. The authors state the GPT-2 experiments are shared with a companion study; the Qwen split training audit is cited from the companion study.
   • Data: the main experiment is fixed 32-token WikiText-2; each round has 32 public calibration, 128 training, and 256 held-out evaluation documents. There are also source code, templated clinical notes, pilot WikiText-2, new documents from the WikiText-103 training split, and 64/128-token text and 32-token dialogue.
   • Seeds and main test: ten independent seeds are fine-tuned with AdamW and then frozen, then attacked with two attack seeds. The main test is pre-fixed: on layer 6, public layer, the lower end of the 95% interval of the token gain of joint minus activation-only must be above 2 percentage points. The bootstrap independently resamples training models and documents, 100,000 times, seed 93281. There are also 47 Bonferroni-corrected secondary comparisons.
   • Controls and budgets: activation-only, joint, shuffled-gradient, random-candidate, backward-only, SipIt; the BiSR comparison also includes the archived reference implementation. joint and activation-only align the number of candidates 32 and beam width 2.
   • Metrics and decision: token recovery; exact-document recovery requires every token in the document to be correct. The scorer compares saved predictions with held-out text; the paper uses no LLM review. The defense gate additionally uses excess relative to the majority token in calibration.
   • Scale: 3,200 training measurements, plus 480 explicitly reused activation-only inference measurements, 3,680 in total, on the same held-out set. Each number in Table 1 aggregates 5,120 document-level scores.

   Main results

   According to Table 1 (GPT-2 124M, the same 256 held-out documents, ten training models, two attack seeds):

   | Depth / layer | Act. token | Joint token | Act. whole-document | Joint whole-document | | 3 / Public | 97.39% | 98.29% | 43.13% | 55.51% | | 3 / Trained | 97.81% | 98.72% | 51.41% | 66.13% | | 6 / Public | 94.20% | 97.38% | 13.71% | 37.77% | | 6 / Trained | 94.81% | 98.27% | 20.23% | 58.55% |

   • Main test: the main text writes the token gain as 3.17 percentage points, Appendix S6 writes 3.1714; 95% interval [2.72, 3.64], with the lower end above the pre-fixed 2 percentage points. The authors report the main test passes (Section 5.1).
   • Secondary comparisons: all 47 pre-specified secondary comparisons favor joint after Bonferroni correction, with intervals above 0 (Section 5.1).
   • Authors' interpretation: the change in whole-document recovery is larger than the token gain suggests. Figure 2 places different observations, knowledge, and investments on the same no-defense layer 6 boundary; matched comparisons follow Table 1.

   Corpus, defenses, and compression

   • Corpus: the same inversion, layer 6 public layer, inference, three seeds, 3,072 positions per corpus (Table 2). Source code recovery 64.94% (baseline 20.31%, excess 46.78 percentage points), templated clinical notes 100.00% (baseline 6.25%, excess 93.75), WikiText-2 63.25% (baseline 2.64%, excess 62.79). The excess uses the Wilson upper bound, not the point estimate minus baseline. All three no-defense conditions fail the token gate. The authors state the clinical result describes rule-based synthetic templates, and does not measure clinical records.
   • Two defenses: under secret mixup, token recovery 83.01–90.67%, whole-document at most 1.48%; name span 47.85–66.15% (716 spans, 249 documents), while without defense name span is 67.36–97.44%. Geometric mean perplexity is higher by 31.8% at layer 3 and 26.9% at layer 6. replica median does not change activation; the authors state the gap between token recovery and the no-defense control is within two-thirds of a percentage point. The most affected whole-document setting goes from 43.77% to 36.11% (public layer, layer 3, SipIt 128 steps, S3.1).
   • Compression and duration: the white-box path that knows the compressor has average token recovery of 18.85–68.59% at each depth of the stable compressor, despite compressing 768 dimensions to 64–256; the co-training protocol is 23.52–67.68% (Section 5.2). The six-defense pilot's verdict is unchanged between 20 and 100 steps. At 500 steps, mid-lite pushes the declared inversion that does not know the projection below the majority baseline 4.83–4.96% (recovery 0.04–0.85%), with cross-entropy 7.42–7.75, versus 4.39–4.44 without defense. On longer text, no-defense inversion is 76.56–90.22%, and activation noise with σ=15 is 9.96–55.05%.

   Porting published attacks

   • Consistency: under the single-seam contract, the port and the archived reference implementation have all tokens identical on 3,072 prediction pairs (Section 5.3). The authors state this does not cover the two-cut protocol of the original BiSR paper.
   • Paired comparison: layer 6 public layer, refined BiSR is 93.21%, with a difference from joint of −4.17 percentage points, 95% interval [−4.79, −3.53]. On the trained layer the difference is −0.02, interval [−0.63, 0.58]. Of 15 secondary comparisons, 10 intervals are below 0, 1 is above 0, and 4 cross 0.
   • Table 3: the authors state that no tested configuration is higher in every setting. For example, at layer 3, trained layer, SipIt-128 is 98.89%, joint is 98.72%, and refined BiSR is 97.79%. Activation, SipIt, and joint are controls reused from the main experiment, and each attack uses its own fixed budget.

   Measurement code and field agent

   • Measurement code: 25 development test functions are pre-selected, parameterized into 37 use cases, covering seven groups of properties. All met expectations, with no failures and no skips (Section 5.4, Table S11). When only the text visible to the BiSR scorer is changed, predictions are byte-for-byte identical; handing that text to the predictor is rejected. When the fully correlated document column increases from 2 to 500, the interval remains [0, 1].
   • Field planner: Qwen3-0.6B, with reasoning off. All ten fixed settings produce measurable actions; twenty trials have no policy violations, but all six allowed peer-request trials are blocked. The authors state this only shows the planner and tools are connected and that over-blocking exists, and does not establish general agentic security (Section 6, S3.6).
   • Companion audit: the authors report that the Qwen3-0.6B split training audit recovers the real/decoy split on all 4,096 frames of the main run, while the forward-only gate still passes; structural recovery and text reconstruction are written as different results (Section 6).

   Other ablations and analyses

   • Layer 6 public layer: shuffled-gradient 93.91%, random-candidate 93.89%, backward-only 49.98% (Section 5.1).
   • Perturbing gradient under the same setting: Gaussian noise at 0.1 times document RMS 97.31%, eight-bit quantization 97.33%.
   • Figure 2(c): SipIt-32 token 91.49%, whole-document 1.70%; SipIt-128 token 95.25%, whole-document 20.76%. Except the last tier, public layer is used.
   • secret mixup: numeric span 93.94–96.50% (297 spans, 142 documents), identifier span 90–100% (2 spans, 2 documents). Annotations are fixed before the run, names have AI assistance; the authors state they are not human gold labels (S3.1).
   • Domain experiment inference-row floor is 0.84–1.90 percentage points; training-row floor is written in Appendix S6 as 3.911, 2.603, 2.342 percentage points (n=768).
   • Noise-aware whitening: inference recovery from 21.65% to 13.09%, training recovery from 11.46% to 13.67% (S3.3).

   Collapse this answer

5. 05What are some points that can be further explored?

   The authors write in Section 6 that the strongest quantitative reconstruction comparison is for the stated setting of GPT-2 124M.

   Limitations and future directions noted by the authors

   The paper has no separate Limitations section. The following statements come from Section 6:

   • Scope of quantitative comparison: the authors write that the paper's strongest quantitative reconstruction comparison is for GPT-2 124M, and the stated corpus, lengths, and budgets.
   • Evidence that can be extended later: longer context, new targets, and other attack budgets can extend the evidence through the same pipeline.
   • Protocols cannot be merged: the authors write that collapsing different protocols into a single success rate loses the explanation provided by the corpus baseline, utility measurements, and matched controls.
   • Agent level: the field level shows the planner and tools are connected, and identifies over-blocking; the authors write that it does not establish general agentic security.

   Scope of experimental coverage

   • The main reconstruction experiment is GPT-2 124M, split at layer 3 or layer 6, 32-token WikiText-2, ten training seeds, two attack seeds, 256 held-out documents (Section 4.1, Table 1).
   • The qualification procedure listed in Section 6 covers Qwen3-0.6B, Qwen3-8B, Qwen3.5-27B, Qwen3.5-35B-A3B, Mistral-7B-Instruct, and Mistral-NeMo-12B-Instruct.
   • Defense-related measurements include secret mixup, replica median, PCA and bottleneck compression, added noise, sparsify-10%, mid-lite, santext-30%, at-lite, and activation noise with σ=15 (Section 5, S3).
   • 64 registered modules, with the qualification matrix in Supplementary Table S15; all 37 selected use cases of the measurement code met expectations (Section 5.4). Scoring compares predictions with held-out text; the paper reports no LLM review, and no agreement of span annotation with human gold labels.
   • BiSR consistency uses single-seam and new documents from the WikiText-103 training split, 3,072 prediction pairs in total; the authors write that the original two-cut is a different protocol (S3.4). The clinical notes are templated synthetic text, and the authors state they do not measure clinical records (Section 5.2).

   Collapse this answer

6. 06Summarize the main content of the paper

   The authors report that gradients, corpora and defenses change recovery, and that the evaluation records can be independently re-evaluated.

   LLM-Attacker is a privacy evaluation framework for split and distributed language models, used to record attacker conditions as verifiable experiments.

   • Problem: the client keeps the text, but activation and the gradient returned during training leave the client. If the observation, knowledge, corpus, or metric changes, the privacy conclusion may change; average token scores may also mask whole-document recovery.
   • Approach: use component E to record the model, observation, knowledge, budget, corpus, metrics, and controls; compatible attacks replay the same capture. The token gate compares the measured attack's Wilson upper-bound excess relative to the majority token in calibration text. A defense pass requires repeated runs and a qualified no-defense reference, and untested attacks remain visible.
   • Main result: on GPT-2 124M, layer 6, public layer, 256 held-out WikiText-2 documents, joint's token recovery is 97.38%, and activation-only is 94.20%. The main text writes the gain as 3.17 percentage points, 95% interval [2.72, 3.64] (Section 5.1, Table 1). Under the same setting, whole-document recovery goes from 13.71% to 37.77%.
   • Other measurements: the same inversion on templated clinical notes has token recovery 100.00%, code 64.94%, WikiText-2 63.25% (Table 2). refined BiSR on layer 6 public layer is 93.21%, with a paired difference from joint of −4.17 percentage points (Section 5.3). All 37 selected use cases of the measurement code met expectations.
   • Authors' conclusion: attacker knowledge, the returned gradient, corpus, defenses, and attack configuration substantially change recovery. Every split-model privacy claim should state the attacker, corpus, and scoring rules, report controls, and mark untested attacks as untested.

   Collapse this answer

Related papers