Skip to content

Timing Is Everything: Inferring Temporal Membership in Continuously Updated Models

Timing is Everything: Inferring Temporal Membership in Continually Updated Models

arXiv· arXiv:2611.05781· Jie Fu, Xiangpeng Deng, Yuan Hong, Wendy Hui WangStevens Institute of Technology; University of ConnecticutIndexed by this site · 15 days ago· Originally publishedAttacks

Quick read

Attacks

Based on the paper PDF (AI-generated); refer to the original

The authors propose a black-box attack, TMIA, which uses a BiLSTM to infer from the output trajectories of consecutive model snapshots when a sample entered/left the training set; on Pythia-70M/SST-5, T-F1 reaches 91.34%, outperforming baselines.

Threat modelBlack-box attacker: can query the target model's snapshot sequence f1..fN and observe posterior probability vectors, and by default can access all snapshots; does not know model internal details, update algorithm, or update data; is passive and cannot influence the update process; possesses one or more shadow datasets to train shadow models.

Questions
Continuously updated models expose temporal-dimension membership information about when a sample entered or left the training set; the authors state that existing MIA considers only unidirectional, single updates and compares only adjacent snapshots, and cannot handle scenarios where insertions and deletions coexist and updates repeat.
Methods
The authors propose a black-box attack, TMIA: train a sequence of shadow models aligned with the target's updates, use a BiLSTM with a sliding window to perform sequence labeling on the confidence trajectory (entry point/exit point/no-change point), and then construct valid membership intervals satisfying temporal constraints through three steps.
Experiments and results
On 3 language models and 3 image models, TMIA's T-F1 and I-F1 are both higher than the TS-MIA and UP-MIA baselines; on LLaMA-3B/8B, T-F1 exceeds 84%; when the shadow model and target mismatch in dataset, architecture and update algorithm, performance degradation is limited; the three defenses label-only, dropout and DP reduce attack effectiveness but harm utility.
Limitations and future work
The authors note that future directions include extending to white-box settings, designing defenses specifically targeting temporal membership leakage, and establishing formal privacy guarantees and benchmarks for evolving models (Conclusion). Experiments cover 8 models and 8 datasets, with results averaged over three trials; membership changes occurring between two snapshots are unobservable to a black-box attacker.
Experimental setupPythia-70M、T5 et al. · SST-5、News20 et al. · T-Pre、T-Rec et al.
Evaluated models
Pythia-70MT5GPT-2 SmallResNet-18SimpleCNNMobileNetLLaMA-3BLLaMA-8B
Benchmarks
SST-5News20MNLICIFAR-10SVHNCINIC-10SQuADSimpleQA
Metrics
T-PreT-RecT-F1I-PreI-RecI-F1hit rate
AI summaryResearchers propose a temporal membership inference attack (TMIA) against continuously updated deep learning models that can infer from black-box model outputs whether and when an individual sample participated in training. The method uses a bidirectional LSTM to analyze model outputs across consecutive updates and supports two granularities, timestamps and time intervals; the authors state this is the first systematic study of temporal privacy risks in continuously updated models. Experiments on three language models and three image classification models show that TMIA is highly effective and significantly outperforms existing baselines; an evaluation of three potential defenses reveals a trade-off between privacy protection and model utility.

Researchers propose a temporal membership inference attack (TMIA) against continuously updated deep learning models that can infer from black-box model outputs whether and when an individual sample participated in training. The method uses a bidirectional LSTM to analyze model outputs across consecutive updates and supports two granularities, timestamps and time intervals; the authors state this is the first systematic study of temporal privacy risks in continuously updated models. Experiments on three language models and three image classification models show that TMIA is highly effective and significantly outperforms existing baselines; an evaluation of three potential defenses reveals a trade-off between privacy protection and model utility.

Paper analysis

6 questions; open one to read the full answer

  1. What problem does this paper attempt to solve?

    In continuously updated models, whether an attacker can infer when a sample entered/left the training set (temporal membership inference); existing MIA cannot handle this.

    Core question: in continuously updated deep learning models, how to infer a single sample's temporal membership—that is, at which time points or time intervals the sample participated in training.

    • Scenario and importance: deployed models are repeatedly updated to incorporate new data, delete outdated samples, or satisfy regulatory requirements such as the "right to be forgotten," forming a sequence of model snapshots. The authors argue that inferring when a sample entered/left training can expose additional sensitive information, for example reconstructing a user's health timeline from the time window during which health-related queries were used.
    • Shortcomings of existing methods: the authors point out that existing MIA for updated models considers either only insertions or only deletions (unidirectional assumption); assumes each sample undergoes at most one membership change; and mostly compares only two adjacent snapshots, which the authors say cannot capture temporal dependencies across updates and may infer temporally inconsistent results (e.g., predicting a sample was deleted before it was inserted) (Table 1).
    • Core observation: the authors' empirical analysis (Section 4.1, Pythia-70M/SST-5 and SimpleCNN/CIFAR-10) finds that after insertion a sample's confidence converges quickly and remains stable, whereas after deletion confidence keeps fluctuating; the authors argue this asymmetric dynamic between insertion and deletion can be used to identify update time points and distinguish insertions from deletions.
    • What the paper proposes: formalizes the temporal membership inference problem (at timestamp level and interval level), proposes the black-box attack TMIA (based on bidirectional LSTM), and evaluates three defenses (label-only, dropout, differential privacy) in the continuously updated setting.
    • Threat model (Section 3.2):
      • Goal: given a target sample x, infer its set of temporal membership intervals I(x)={I1,...,Ik}.
      • Knowledge: black-box access to the model snapshot sequence f1..fN, can query arbitrarily and observe posterior probability vectors; by default can access all snapshots (Section 7 evaluates the case of accessing only some snapshots); does not know model parameters, architecture, training hyperparameters, update algorithm, or update data content.
      • Capability: passive attacker, cannot influence the update process (cannot inject samples and then request deletion); possesses one or more shadow datasets, which need not be from the same distribution as the target training data.
      • Victim system: continuously updated classification models (language models and image classification DNNs), as well as the extended large generative language models.
  2. What related research exists?

    Related work falls into three categories: MIA against model updates, MIA against machine unlearning, and static model MIA and defenses; the authors state that all of them cover only unidirectional single updates.

    MIA against model updates

    • Jagielski et al. [26]—the authors call it the first MIA against updated models; it considers single-update and multi-update settings and proposes two attacks, membership inference (determining whether a sample is in the update set at a given time point) and entry inference (determining at which epoch a sample entered the dataset); its delta attack is the source of this paper's UP-MIA baseline. The authors point out that it considers only data insertion.
    • Michel et al. [35]—proposes the Sequential Membership Inference (SeMI) attack, which uses the model sequence and injects canaries at controlled insertion times, derives the optimal SeMI in the form of a likelihood ratio (LR) test, and designs an LR white-box attack and a loss-based black-box attack. The authors point out that it likewise considers only insertion.
    • The authors state that the insertion-only setting of these two papers is a special case of this paper's scenario (insertions and deletions coexisting), and therefore does not directly apply.

    MIA against machine unlearning (MU) models

    • Chen et al. [12], Hayes et al. [21], Naderloui et al. [37], etc.—use the difference between the original model and the unlearned model to infer whether the target sample belongs to the deleted set. The authors point out that these works focus only on inference of a single deletion event and do not apply to the more general scenario where insertions and deletions occur continuously.

    Static model MIA and sequence modeling tools

    • Shokri et al. [52] propose membership inference attacks against machine learning models; Song and Mittal [54]'s entropy-based MIA is the source of this paper's TS-MIA baseline; Yeom et al. [59], Salem et al. [49], etc. train static shadow models to approximate a single fixed target model, and the authors point out that this paper's update-aligned shadow model training is more complex than static shadow model training.
    • Tools: LSTM [23] and BiLSTM [20] are used for sequence labeling; CUSUM [30] change point detection is used as the CP-MIA straw-man baseline; sliding-window LSTM methods [7, 42, 50] are introduced to emphasize local temporal patterns.

    Baselines and benchmarks: the comparison baselines are TS-MIA (entropy-based MIA [54], applied independently snapshot by snapshot) and UP-MIA (delta attack [26], comparing the loss difference between adjacent snapshots); because CP-MIA assumes the number of membership changes is known, the authors did not include it in the formal comparison (Section 6.1 footnote). Benchmarks are the text datasets SST-5, News20, MNLI and the image datasets CIFAR-10, SVHN, CINIC-10; extended experiments use SQuAD and SimpleQA.

    Positioning: the authors call this paper the first systematic study ("the first systematic study") of temporal privacy risks in continuously updated models; its difference from prior work is that it supports both insertions and deletions, supports repeated updates, and uses the full snapshot sequence rather than adjacent snapshot pairs (Table 1).

  3. How does the paper address this problem?

    TMIA has three steps: train a sequence of shadow models aligned with the target's updates, use a sliding-window BiLSTM to label entry/exit points, and then construct valid membership intervals that satisfy temporal constraints.

    Overall approach: treat the target sample's outputs across model snapshots (confidence trajectory) as a time series, use the TMIA framework to convert it into a sequence labeling problem, identify entry points and exit points, then assemble them into valid temporal membership intervals.

    Problem formalization and metrics

    • Temporal membership: sample x is a member of snapshot fi if and only if x belongs to the initial training set and has never been updated, or the most recent update involving x as of fi is an insertion (Definition 1). A membership interval [t_start, t_end] is bounded by an entry point and an exit point; a sample may enter and exit multiple times, forming multiple disjoint intervals.
    • Two-level metrics: timestamp-level precision/recall T-Pre(x) = |T(x)∩T*(x)| / |T(x)|, T-Rec(x) = |T(x)∩T*(x)| / |T*(x)|, and their harmonic mean T-F1; interval-level I-Pre, I-Rec and I-F1 are defined by the overlap ratio between predicted intervals and true intervals, and the authors note that interval-level metrics penalize fragmented predictions.
    • Attack definition: learn a mapping A: (x, f1(x), f2(x), ..., fN(x)) → {I1, I2, ..., Ik}, i.e., infer the set of membership intervals from a sample and its outputs on each snapshot.

    Training update-aligned shadow models

    • The attacker trains an initial shadow model f0^S with a shadow dataset D^S, then sequentially applies shadow update sets Δi^S to obtain a shadow snapshot sequence {f1^S,...,fN^S}, simulating the evolution of the target model.
    • The shadow model's architecture, data distribution, hyperparameters and update algorithm may all differ from the target; the authors state that the impact of such mismatch is evaluated in Section 6.3.

    Sequence labeling: sliding-window BiLSTM

    • A sample set X^S is uniformly sampled from each of the shadow model's sets; for each sample, the confidence of outputs at each snapshot is collected to form a univariate time series s=(s1,...,sN), and each timestamp is labeled as N (non-change point), E (entry point) or X (exit point) according to the true update records.
    • The labeling model is a BiLSTM with a sliding window: for each timestamp, a window of length 2w+1 is taken (out-of-bounds positions are padded with boundary values), and the hidden states of the forward/backward LSTMs are reset at window boundaries, so that the hidden state at ti depends only on inputs within the window; the bidirectional hidden states are concatenated and passed through a linear layer and softmax to output three classes of labels, trained with cross-entropy. The authors' rationale: traditional LSTMs emphasize global long-range dependencies, and distant context may become noise, whereas the sliding window emphasizes local patterns near entry/exit points.
    • Attack model parameters (Appendix B.4): BiLSTM feature dimension 32, followed by a 32-neuron ReLU hidden layer and a 3-neuron softmax output layer; 3,000 shadow samples are randomly selected for training and 1,200 target samples for evaluation.

    Constructing valid membership intervals

    • Independent per-timestamp labeling may produce temporal inconsistencies (e.g., an exit point preceding an entry point), so a valid membership interval is defined to require: correct ordering, injective entry-exit matching, and non-overlapping intervals (Definition 4).
    • Three-step construction: 1) discard N labels and group consecutive same-label timestamps into maximal groups; 2) take the timestamp with the highest confidence in each group as the representative; 3) process the representative points in temporal order, matching each exit point with the immediately adjacent unmatched entry point to form a closed interval; if the sequence ends with an unmatched entry point, the interval extends to the last snapshot, and if it begins with an exit point, an implicit entry point t0 is used.
    • Theorem 5 states that this three-step process guarantees returning valid membership intervals; the proof is in Appendix A.

    Success determination

    • Attack effectiveness is measured by T-F1 and I-F1; the authors explain that AUC is not applicable because the attack directly outputs intervals rather than probabilistic decisions. Appendix D additionally defines the hit rate of entry/exit points within a tolerance window θ to evaluate localization precision.
  4. What experiments did the paper conduct?

    On 6 classification models×6 datasets and 2 LLMs×2 QA datasets, TMIA's T-F1/I-F1 is higher than both baselines across the board, and mismatch, quantization, partial observation, and three defenses were evaluated.

    Experimental Setup

    • Models evaluated: language models Pythia-70M, T5, GPT-2 Small; image classification ResNet-18, SimpleCNN, MobileNet; extended experiments use LLaMA-3B, LLaMA-8B.
    • Datasets: text SST-5, News20, MNLI; image CIFAR-10, SVHN, CINIC-10; generative experiments use SQuAD (about 100K QA pairs) and SimpleQA (about 4K QA pairs).
    • Update simulation: LM simulates 80 update steps, DNN 60 steps, each step containing both insertions and deletions; update size is 1% of the training set (SST-5, News20, CIFAR-10, SVHN), 0.5% (MNLI), 0.1% (CINIC-10); samples can enter and leave multiple times, and 31%–55% of updated samples in each dataset have at least two membership intervals. Update algorithms: LM uses fine-tuning, GA, NPO; DNN uses fine-tuning, GA, SCRUB; LLaMA uses LoRA fine-tuning, 40 timestamps, 1% per step.
    • Baselines: TS-MIA (entropy-based MIA applied snapshot by snapshot), UP-MIA (delta attack, threshold calibrated with shadow models); on generative models the baselines are switched to PPL versions.
    • Metrics and determination: T-Pre/T-Rec/T-F1 and I-Pre/I-Rec/I-F1 (definitions in Section 3.3); attack features are confidence trajectories, and the LLaMA experiments use perplexity (PPL) sequences instead.
    • Sample sizes and repetitions: the attack model is trained on 3,000 shadow samples and evaluated on 1,200 target samples (Appendix B.4); all results are averages of three trials; among evaluation samples, about 1% are members throughout and 99% undergo membership changes (Appendix B.3).

    Main Results

    Excerpts from Table 2 (timestamp-level T-F1, %) and Table 3 (interval-level I-F1, %):

    Swipe horizontally to read the table

    ModelDatasetTMIA T-F1TS-MIA T-F1UP-MIA T-F1TMIA I-F1
    Pythia-70MSST-591.3477.5957.8884.16
    Pythia-70MNews2083.5566.6039.0777.24
    T5MNLI91.1075.9554.6471.17
    GPT-2News2083.7565.0841.0376.37
    ResNet-18SVHN72.7456.5543.2065.63
    SimpleCNNCIFAR1078.2674.8547.7463.64
    MobileNetCINIC1077.3074.5066.8067.13

    (The full 18 model×dataset combinations are in Table 2/3; the remaining 11 rows are omitted here, of which the lowest are SimpleCNN/SVHN with T-F1 71.02 and I-F1 63.85.)

    • The authors state that TMIA has the highest T-F1 and I-F1 among the three on all 18 settings, with I-F1 improving by at least 21.82% relative; on the three LMs, TMIA's T-Pre and T-Rec are both ≥ 80.51%.
    • The authors point out that TS-MIA achieves high recall by predicting entry points at almost all timestamps (T-Rec often exceeds 85%) but has low precision (T-Pre mostly below 70%); UP-MIA has both low precision and low recall.
    • The authors point out that interval level is harder than timestamp level (boundaries must be located accurately and continuity maintained); on vision models the absolute I-F1 is lower, but TMIA is still highest.

    Shadow Model Mismatch (Section 6.3, Figure 5/9)

    • What was tested: with the target fixed as Pythia-70M/SST-5 etc., the shadow and target were made to mismatch on dataset, architecture, update algorithm, and update size respectively, reporting T-F1 (I-F1 in Appendix E).
    • Results: with dataset mismatch, T-F1 drops by at most 4.95%; with architecture mismatch, by at most 2.1%; with update algorithm mismatch, T-F1 remains at least 88.33%; with update size mismatch, when the shadow size is no larger than the target, T-F1/I-F1 drop by at most 1.91%/4.04%, but when the shadow uses 5% and the target uses 0.5% or 1%, T-F1 drops by at most 18.09%.
    • Authors' interpretation: TMIA captures the basic behavioral changes caused by updates and can transfer across datasets, architectures, algorithms, and update sizes.

    Extension to Large Generative Models (Section 6.5, Table 4/11)

    • What was tested: LLaMA-3B/8B on SQuAD and SimpleQA with 40 LoRA update steps, using PPL sequences instead of confidence.
    • Results: TMIA's T-Pre is 95.61%–99.48%, T-F1 all exceed 84% (highest LLaMA-3B/SimpleQA 95.25%); I-F1 reaches up to 95.01% (LLaMA-3B/SimpleQA, Table 11), and the two baselines' I-F1 is clearly lower.
    • Authors' interpretation: TMIA remains effective on large generative language models.

    Robustness in Realistic Scenarios (Section 7)

    • Correlated updates (Table 5, SST-5/Pythia-70M): when updated samples come from the same sentiment class, T-F1/I-F1 change by at most 0.49%/0.66% compared with the uncorrelated baseline (90.14/80.03); the attacker's shadow model is trained only on uncorrelated updates.
    • Output quantization (Table 6, Pythia-70M): 8-bit quantization reduces T-F1/I-F1 by 9.82%/7.51%, 4-bit by 26.37%/17.53% (under 4-bit, T-F1 63.27%, I-F1 62.50%); it is assumed the attacker knows the quantization algorithm and configuration and applies the same quantization to the shadow model.
    • Partial snapshot observation (Table 7, Pythia-70M): when only 50% of snapshots are accessed, T-F1 72.82%, I-F1 64.66%; when only 25% are accessed, T-F1 48.48%, I-F1 36.47%. The authors state that periodic access still constitutes a privacy threat.

    Defense Evaluation (Section 8, Table 8)

    • What was tested: three defenses for static models adapted to the continuously updated setting—label-only output (using a 0/1 correctness indicator instead of confidence), dropout (p=30%/50%, enabled in both training and inference), and DP-SGD (ε_total∈{10,30}, δ=5×10⁻⁴).
    • Results (SST-5/Pythia-70M): label-only reduces T-F1 from 91.34 to 87.99 and I-F1 from 84.16 to 74.85, with test accuracy almost unchanged (43.26→43.25); DP ε=10 reduces T-F1 to 17.47 and I-F1 to 5.20, with test accuracy dropping to 38.43; dropout p=30% gives T-F1 75.36, I-F1 73.22, test accuracy 40.13. The trend is the same on CIFAR10/SimpleCNN.
    • Authors' interpretation: DP provides the strongest protection but the largest utility loss, label-only almost preserves utility but provides weak protection, and the authors consider dropout the best balance between privacy and utility.

    Other Ablations and Analyses

    • Sliding window size (Figure 6a, Pythia-70M/SST-5): T-F1 is best at window 7, I-F1 is highest at 11, and further increases cause a slight decline.
    • Update size per timestamp 0.1%→10% (Figure 6b): T-F1 and I-F1 rise accordingly.
    • Number of updates per sample 1→5 (Figure 6c): T-F1 drops by 1.3%, I-F1 drops by 1.98%.
    • Increasing the proportion of membership duration (Figure 6d): T-F1 and I-F1 keep rising.
    • Entry/exit point hit rate (Appendix D, Figure 8): at θ=0, TMIA's entry point hit rate is 81.04% (Pythia-70M/SST-5) vs TS-MIA's 63.75%, and 71.00% (ResNet-18/CIFAR-10) vs 33.33%.
    • Negative results: when partial snapshot observation drops to 25%, the attack effect weakens markedly (Table 7); when the shadow update size is 5% against a target of 0.5%/1%, T-F1 drops by at most 18.09% (Figure 5d).
  5. What are some points that can be further explored?

    The authors list three future directions in the Conclusion; experiments cover 8 models and 8 datasets, and in the black-box setting changes between snapshots are unobservable.

    Limitations and Future Directions Noted by the Authors

    • White-box extension (Conclusion): this paper focuses on black-box attacks; extending to the white-box setting and studying how model-internal information such as gradients interacts with different update strategies is an open problem.
    • Dedicated defenses (Conclusion): the authors state that although common defenses were evaluated, designing methods specifically to mitigate temporal membership leakage is largely unexplored.
    • Formal guarantees and benchmarks (Conclusion): the authors believe formal privacy guarantees and benchmarks for evolving models that account for temporal dynamics are needed to define and audit privacy.
    • Changes between snapshots are unobservable (Section 3.2): insertions/deletions that occur entirely between two queried snapshots leave no directly observable evidence and cannot be recovered by any black-box attacker; the authors state this is a fundamental limitation of the threat model rather than of the attack itself.

    Scope of Experimental Coverage

    • 8 models evaluated in total: Pythia-70M, T5, GPT-2 Small, ResNet-18, SimpleCNN, MobileNet (classification), LLaMA-3B, LLaMA-8B (generative, using PPL features).
    • 8 datasets in total: SST-5, News20, MNLI, CIFAR-10, SVHN, CINIC-10, SQuAD, SimpleQA; update steps are 80 (LM), 60 (DNN), 40 (LLaMA).
    • Update algorithms cover fine-tuning, GA, NPO (LM) and fine-tuning, GA, SCRUB (DNN); the LLaMA experiments use only LoRA fine-tuning.
    • Defense evaluation covers three types—label-only, dropout, DP-SGD—on two settings, SST-5/Pythia-70M and CIFAR10/SimpleCNN (Table 8).
    • All results are averages of three trials (Section 6); the attack model is trained on 3,000 shadow samples and evaluated on 1,200 target samples (Appendix B.4); the paper does not report the number of queries or computational cost of the attack.
  6. Summarize the main content of the paper

    TMIA uses a sliding-window BiLSTM to infer a sample's temporal membership intervals from the output trajectories of snapshots, outperforming baselines on 8 models, and all three defenses are effective at a utility cost.

    An attack paper on the privacy of continuously updated models: the authors formalize the concept of "temporal membership" and propose a black-box attack, TMIA, that infers when a sample enters and leaves the model's training set.

    • Problem: deployed models are repeatedly updated to form a snapshot sequence, and samples may be inserted and deleted multiple times; the authors state that existing MIAs against updated models consider only unidirectional, single changes and compare only adjacent snapshots, and cannot reconstruct membership history.
    • Method: the attacker trains a sequence of shadow models aligned with the target's updates, uses a sliding-window BiLSTM to label confidence trajectories as entry points/exit points/non-change points, then constructs valid membership intervals satisfying ordering, matching and non-overlap constraints via a three-step process (Theorem 5 guarantees validity); for generative models, PPL trajectories are used instead.
    • Main results: on 3 LMs and 3 image models across 6 datasets, TMIA's T-F1 and I-F1 are higher than TS-MIA and UP-MIA in all 18 settings (Table 2/3), e.g., T-F1 91.34% and I-F1 84.16% on Pythia-70M/SST-5; the authors state the relative improvement in I-F1 is at least 21.82%.
    • Extension and robustness: T-F1 exceeds 84% on both LLaMA-3B/8B (Table 4); when the shadow model and target mismatch in dataset, architecture and update algorithm, T-F1 drops by at most 4.95%, 2.1% and still ≥ 88.33% respectively; under 4-bit quantization T-F1 is 63.27%; with access to only 50% of snapshots T-F1 is 72.82%.
    • Defenses: label-only, dropout and DP all reduce attack effectiveness; DP ε=10 lowers T-F1 on SST-5/Pythia-70M to 17.47% but reduces test accuracy from 43.26 to 38.43; the authors consider dropout the best balance, and there is a privacy-utility trade-off.
    • Authors' conclusion: the temporal dimension leaks sensitive information beyond "whether it was used"; future directions include white-box extension, dedicated defenses, and formal privacy guarantees and benchmarks for evolving models.

Related papers

Show 2 more papers
Read originalarxiv.org