SoK: How robust are deep neural network watermarking for image classification? (Extended version)

1. 01What problem does this paper attempt to solve?

   The authors systematically evaluate whether DNN watermarking schemes claimed to be robust can withstand a large number of removal attacks under a unified adversarial model.

   Core question: are DNN watermarking schemes claimed to be robust actually robust under a unified, comprehensive set of removal attacks?

   • Scenario and motivation: DNN watermarks are used for model provenance verification; a model provider (e.g., MLaaS) wants to detect a (stolen) surrogate model that a user derived from the source model and publicly deployed; robustness requires that an attacker cannot derive a surrogate model that does not retain the watermark.
   • Existing gaps: the authors state that many schemes claim robustness but are verified in isolation on relatively small attack sets, and most schemes do not state whether their robustness definition covers model extraction attacks; existing attacks have broken some schemes' security claims, but it is unclear whether this generalizes to other watermarks.
   • Core approach: survey 29 methods (known removal attacks, methods that can derive surrogate models but were not evaluated as removal attacks, and new attacks), propose a taxonomy of watermarking schemes and removal attacks, a unified adversarial model, and implement the open-source toolbox WRT.
   • Main findings: the authors state that no evaluated scheme is robust in practice; schemes are broken by adaptive attacks and by surrogate model derivation methods not previously evaluated as removal attacks; but no single attack can remove all watermarks, while combined attacks can.
   • Threat model:
     • Goal: derive a surrogate model that does not retain the watermark and has test accuracy close to the source model; a secondary goal is to reduce attack resource consumption (e.g., runtime).
     • Knowledge: white-box (access to all source model parameters) or black-box (API only); the attacker knows the source model architecture.
     • Data: Labeled (same distribution, at most one third labeled), Domain (same distribution, unlabeled), Transfer (different distribution, labeled; Open Images used in experiments).
     • Victim system: an image classification model trained on CIFAR-10 / ImageNet with an embedded watermark.

   Collapse this answer

2. 02What related research exists?

   Related work falls into four categories: model extraction attacks, 11 watermarking schemes, known removal attacks, and surrogate model derivation methods not evaluated as removal attacks.

   Model extraction (model stealing) attacks

   • Krishna et al. [8] — the authors cite their conclusion: even for large models such as BERT, a high-fidelity surrogate can be derived with limited domain data, at a cost far lower than retraining.
   • Papernot et al. [10] — list confidentiality (protecting model parameters from theft) as one of the core principles of machine learning security and privacy.
   • Tramèr et al. [36], Jagielski et al. [11], Carlini et al. [12], Atli et al. [13] — extract models via prediction APIs or cryptanalytic methods; the authors cite them to show that preventing model stealing is very difficult.

   DNN watermarking schemes (subjects of this paper's evaluation, 11 in total)

   • Model-agnostic: Adi [16] (uses out-of-distribution abstract images as keys), Zhang [17]'s three schemes Content/Noise/Unrelated (differing only in key selection).
   • Model-related: Frontier-Stitching [26] and Blackmarks [24] (based on adversarial examples), Jia [25] (entangles features using soft nearest neighbor loss).
   • Parameter encoding: Uchida [14] (embeds in convolutional layer weights), DeepMarks [23] (similar to Uchida, extracted via correlation), DeepSigns [15] (embeds in hidden-layer activations); active: DAWN [27] (randomly alters a small number of predictions at inference time).

   Known removal attacks and surrogate model derivation methods

   • Known removal attacks: weight pruning [21], fine-tuning / overwriting [14], Neural Cleanse [37], Neural Laundering [35], Fine-Pruning [38], Regularization [18], etc.
   • Methods not evaluated as removal attacks: knowledge distillation [22], Knockoff Nets [40], transfer learning [39], adversarial training [41], input reconstruction [46], JPEG compression [44], feature squeezing [43], etc.
   • The authors point out that [14], [16], [23] restrict the attacker's computational resources in their threat models, an assumption the authors consider unrealistic; attacks designed against model-agnostic watermarks such as [18], [35] often only lower watermark accuracy in experiments rather than removing the watermark.

   Baselines and benchmarks: this paper does not propose new schemes compared against baselines, but rather evaluates the above 11 schemes and 29 attacks against each other under a unified framework; the datasets are CIFAR-10 [28] and ImageNet [29], with Open Images [49] as the transfer dataset.

   Positioning: the authors state that this paper is the first work to conduct a unified, systematic empirical evaluation of watermark robustness claims (the original wording is "there is no systematic, empirical evaluation", and this paper fills this gap), and propose a taxonomy, a unified adversarial model, and an open-source toolbox.

   Collapse this answer

3. 03How does the paper address this problem?

   Using a unified adversarial model and Nash equilibrium, perform parameter ablation evaluation of 11 schemes × 29 attacks in the WRT toolbox.

   Overall approach: classify watermarking schemes and removal attacks separately, and under a unified adversarial model let the defender and attacker each choose the optimum from their parameter sets (Nash equilibrium), thereby determining whether a scheme is robust; all methods are implemented in the open-source toolbox WRT (PyTorch backend).

   Taxonomy

   • Four categories of watermarking schemes (Figure 1): model-related (key depends on the model, e.g., adversarial examples), model-agnostic (key does not depend on the model; the watermark is a backdoor), active (embedded at inference time, only defends against black-box attackers), parameter encoding (embedded in weights or hidden-layer activations, verification requires white-box).
   • Three categories of removal attacks: input preprocessing (requires white-box), model modification (requires white-box, e.g., fine-tuning, pruning), model extraction (mostly black-box, training a different surrogate model).
   • Zero-bit and multi-bit: zero-bit watermarks encode only existence, multi-bit watermarks can encode a multi-bit message; Table II lists each scheme's category, verification method (black-box/white-box), and capacity.

   Unified Adversarial Model

   • Security game: the defender embeds the watermark M̂ ← Embed(T, m, M), the attacker derives M1 ← A(M̂); a sample is drawn at random from {M0, M1} and given to the defender for verification; if the defender's distinguishing success rate does not exceed 0.5 + ε, the attacker wins.
   • Capability dimensions: deployment (white-box/black-box), dataset (Labeled/Domain/Transfer), speed (computational resources are assumed unlimited, only runtime is measured; classified as fast relative to training from scratch <25%、中 25–75%、慢 >75%, based on ImageNet).
   • Table III summarizes each attack's category and capabilities, e.g., distillation requires white-box, while retraining and transfer learning require only black-box.

   New Attack Designs

   • Weight shifting: an adaptive attack against parameter-encoding schemes (Uchida, DeepMarks). Exploiting the weakness that "if all convolutional filters are negated, watermark accuracy is zero", each filter is softly perturbed along the inverse-mean direction with added Gaussian noise, then fine-tuned to restore accuracy: S(W; λ₁, λ₂)_i = W_i − (λ₁)/n∑_j W_j − λ₂ A. Parameters: λ1=1.5, λ2=1.0 for CIFAR-10, λ1=1.3, λ2=0 for ImageNet; fine-tune for 10 / 5 epochs.
   • Smooth retraining: an adaptive attack against the active scheme DAWN. Query the source model with multiple randomly affine-transformed versions of the same image, take the mean of the returned labels as that image's label to train the surrogate; the experiment uses n=3 queries.
   • Feature permutation: an adaptive attack against DeepSigns, randomly permuting hidden-layer neurons (model function unchanged).

   Metrics and Decision

   • Quantities: embedding loss (test accuracy difference between unwatermarked and watermarked models), stealing loss (test accuracy difference between the watermarked source model and the surrogate model), watermark accuracy (match rate between extracted message and embedded message), runtime (single Tesla P100 GPU).
   • Decision thresholds: the papers of the schemes do not give a threshold derivation method; the authors estimate them empirically — generate 100 keys of length N=100, compute watermark accuracy on 30 (CIFAR-10) / 20 (ImageNet) unwatermarked models, and take the cumulative normal distribution threshold at p-value 0.05 (Table IV); then linearly scale watermark accuracy to a unified threshold θ0=0.5 for cross-scheme comparison.
   • Success criterion: the surrogate model's watermark accuracy is below the decision threshold, and stealing loss does not exceed 5 percentage points (regarded as well-trained).

   Nash Equilibrium

   • For each scheme and attack, perform multiple parameter ablations (Appendix XII, XIII) and construct a payoff matrix: the payoff is the surrogate model's test accuracy when the attack succeeds, otherwise zero.
   • The defender minimizes the payoff and the attacker maximizes it; take the parameter combination at the Nash equilibrium (d^∗, a^∗) = argmin_i(argmax_j V[i,j]), to show that attack success stems from the scheme's own weakness rather than inappropriate parameter choice.

   Collapse this answer

4. 04What experiments did the paper conduct?

   Evaluating the robustness of 11 schemes against 29 attacks on CIFAR-10 and ImageNet; the authors state that no scheme is robust and that combined attacks can remove all watermarks.

   Experimental Setup

   • Subjects under test: 11 watermarking schemes — Adi, Content, Noise, Unrelated, Jia, Frontier Stitching, Blackmarks, DeepMarks, DeepSigns, Uchida, DAWN (Table II).
   • Model architectures: Wide ResNet 28x10 for CIFAR-10, ResNet-50 for ImageNet; DenseNet-121 for cross-architecture retraining; the attacker knows the source model architecture. Source model test accuracy: CIFAR-10 94.20%, ImageNet 75.48% (Appendix XIV).
   • Datasets: CIFAR-10 (50k training images, 10 classes), ImageNet (1.28M training images, 1k classes); transfer dataset Open Images (using a 1.7M image subset). Model-modification attackers receive one third of the training data; model-extraction attackers receive all training data (unlabeled, using source model predictions as substitutes).
   • Attack set: 29 methods, divided into three categories: input preprocessing, model modification, and model extraction (Table III), including the weight shifting, smooth retraining, feature permutation, and cross-architecture retraining proposed in this paper.
   • Metrics and decision: watermark accuracy (scaled to θ0=0.5 according to the Table IV threshold), stealing loss ≤ 5 percentage points is considered well-trained; decision thresholds estimated using 100 keys × 30/20 unwatermarked models, p-value 0.05.
   • Hardware: a single Tesla P100 GPU; training from scratch on CIFAR-10 / ImageNet takes about 1h / 100h.

   Main Results

   Robustness overview (Table V, X=robust, 7=not robust, format is CIFAR-10/ImageNet, excerpt):

   Swipe horizontally to read the table

   Attack	Content	Jia	DeepMarks	DeepSigns	Uchida	DAWN
   Input Smoothing	X/X	X/X	X/X	X/X	X/X	X/X
   Regularization	7/X	7/X	X/X	X/X	X/X	X/7
   Feature Permutation	X/X	X/X	X/X	7/7	X/X	X/X
   Weight Shifting	7/X	X/X	7/7	7/7	7/7	X/X
   Retraining	7/7	X/X	7/7	7/7	7/7	X/7
   Transfer Learning	7/7	7/7	7/7	7/7	7/7	X/7
   Smooth Retraining	-	-	-	-	-	7/7

   (According to Table V; the five columns Noise, Unrelated, Adi, Frontier Stitching, Blackmarks and the rows Knockoff Nets, Cross-Architecture Retraining, etc. are omitted; "-" indicates that the attack was not executed against that scheme.)

   • No scheme is robust: the authors state that under Nash equilibrium parameters, no scheme can withstand all attacks (in Figure 3b, 3e the best-attack watermark accuracy of all schemes is below θ0=0.5).
   • No single attack removes all watermarks: transfer learning is the dominant attack on CIFAR-10 (removes all 11 watermarks), but on ImageNet there is no dominant attack (Noise and Blackmarks are robust to transfer learning on ImageNet).
   • Combined attacks: the authors state that the combined attack of transfer learning + label smoothing removes all 11 watermarks on CIFAR-10 and ImageNet.

   Effectiveness of Attack Categories (Figure 4 radar chart)

   • Model extraction is most effective: the authors state that almost all schemes are not resistant to model extraction attacks on CIFAR-10 and ImageNet; transfer learning is the most effective, requiring only about 40% (CIFAR-10) / about 25% (ImageNet) of the time to train from scratch.
   • Model modification is next: it can remove most watermarks (except Jia on ImageNet); Uchida is only removed by the adaptive weight shifting.
   • Input preprocessing is weakest: in most cases it cannot remove watermarks and only lowers watermark accuracy; DeepSigns is an exception due to feature permutation.

   Data Volume and Runtime

   • Data volume ablation (Figure 3a, 3d): on CIFAR-10, retraining requires at least about 20k unlabeled samples to successfully attack, while transfer learning requires only about 5k; on ImageNet, a successful attack requires at least 500k samples, and beyond 250k the gap between the two approaches approaches zero.
   • Runtime (Figure 2): model-extraction attacks are overall far slower than model-modification attacks; Knockoff Nets is the slowest; embedding time is short for all (on CIFAR-10, about 20% of training time for model-agnostic schemes and about 9% for parameter-encoding schemes), based on which the authors consider all evaluated schemes efficient.
   • Cost estimate: the authors state that on ImageNet, training ResNet-50 from scratch takes about 100 hours and about 43 USD, while transfer learning takes about 23 hours and about 10 USD (at 0.43 USD/GPU hour).

   Other Ablations and Analyses

   • Adaptive attacks: smooth retraining is the only attack on CIFAR-10 that can remove DAWN; feature permutation is only effective against DeepSigns; weight shifting is the only model-modification attack on CIFAR-10 and ImageNet that can remove Uchida.
   • embedding loss: about 1 percentage point on both datasets, with the exception of DeepSigns exceeding 3 percentage points on ImageNet; DeepMarks has the lowest embedding loss on both datasets (Figure 2c, 2f).
   • Decision thresholds: the thresholds of Content, Noise, Frontier Stitching, Blackmarks are affected by parameter choice; Table IV gives the maximum computed values (details in Appendix XV).
   • DeepSigns reproduction issue: the authors could not stably embed on ImageNet with m>1 Gaussian distributions, and only reproduced it by switching to m=1 Gaussian, n=100 bits, and alternating training (Appendix XII).
   • Single-class forgetting: the authors observe that when schemes such as Content and Jia embed, the source model may forget a single class; on ImageNet the total test accuracy then drops by only about 0.1%, and they recommend evaluating single-class accuracy degradation.

   Collapse this answer

5. 05What are some points that can be further explored?

   The authors provide robustness evaluation guidelines and note that the usability of threshold estimation is limited; the experiments cover only two image classification datasets.

   Limitations and Future Directions Noted by the Authors

   • Limited usability of the decision threshold method (Section IX-A): the empirical threshold estimation proposed by the authors requires training many unwatermarked models on CIFAR-10 and ImageNet, "hence the usability is limited".
   • Evaluation guidelines (Section IX-A): recommend experiments on (i) small datasets, (ii) datasets with large input sizes, (iii) datasets with many classes; list the amount of data and number of labels used by the attack; give a threshold derivation method; disclose all parameters and evaluate under Nash equilibrium; evaluate single-class accuracy degradation; report embedding/removal runtime relative to training from scratch.
   • Stricter definition and evaluation of robustness (Section IX-B): the authors believe DNN watermark robustness needs to be defined and evaluated more strictly; many prior works target weak attackers that do not adapt their attacks, or restrict the attacker's computational resources and availability of pretrained models.
   • Does not mean watermarking is impossible (Section IX-B): the authors state that this paper does not imply DNN watermarking is infeasible, and point out that the fingerprinting scheme [60] shows promising results.
   • Combined attacks unexplored (Section VIII-G): the authors state that the threat of combined attacks to watermark robustness has not yet been explored, and this paper shows they can pose a threat.
   • WRT open source (Section VIII-A): the authors plan to release WRT as open-source code ("will be made available"), so that future schemes can reuse its attacker models for evaluation.

   Experimental Coverage

   • 11 watermarking schemes under test and 29 attack methods, all listed in Table II and Table III.
   • The datasets are two image classification datasets, CIFAR-10 and ImageNet, with Open Images as the transfer dataset; the model architectures are Wide ResNet 28x10 and ResNet-50, with DenseNet-121 for cross-architecture experiments.
   • Decision threshold estimation uses 100 keys and 30 (CIFAR-10) / 20 (ImageNet) unwatermarked models; threshold estimation additionally used 20 pretrained torchvision models (Appendix XV).
   • The paper does not report complete numerical results for each "scheme × attack" combination (stating they will be released via interactive charts); the main text gives only summary conclusions.
   • The paper does not report the number of queries for attacks; the number of queries for smooth retraining is only stated as n=3.

   Collapse this answer

6. 06Summarize the main content of the paper

   After systematic evaluation, the authors state: none of the 11 DNN watermarking schemes is robust, model extraction attacks are the most effective, and combined attacks can remove all watermarks.

   This is a SoK paper systematically evaluating the robustness of DNN watermarking (short version published at IEEE S&P'22).

   • Problem: DNN watermarking is used to detect model stealing, but the robustness claims of each scheme are verified in isolation under a small number of attacks, lacking unified evaluation, making practical deployment hard to trust.
   • Method: Proposes a taxonomy of four types of watermarking schemes (model-dependent/independent, parameter encoding, active) and three types of removal attacks (input preprocessing, model modification, model extraction); a unified adversarial model covering white-box/black-box and three data availability settings; uses Nash equilibrium over parameter ablations to select optimal parameters for both sides; decision thresholds are empirically estimated using 100 keys and 20–30 unwatermarked models (p=0.05); all methods are implemented in the open-source toolbox WRT.
   • New attacks: weight shifting (against Uchida, DeepMarks), smooth retraining (against DAWN), feature permutation (against DeepSigns).
   • Main results:
     • The authors state that none of the 11 schemes can withstand all attacks on CIFAR-10 and ImageNet (Figure 3, Table V).
     • Model extraction attacks are the most effective; transfer learning removes all 11 watermarks on CIFAR-10, but there is no dominant attack on ImageNet.
     • The combined attack of transfer learning + label smoothing removes all 11 watermarks on both datasets.
     • Data thresholds for successful attacks: on CIFAR-10, transfer learning requires about 5k unlabeled samples and retraining about 20k; ImageNet requires at least 500k.
     • Cost: the authors estimate the transfer learning attack on ImageNet takes about 23 hours and about 10 USD.
   • Conclusion and implications: The authors argue that robustness should be evaluated under a broader set of attacks and more realistic adversarial models, and provide evaluation guidelines (multiple datasets, threshold derivation, parameter ablation, single-class accuracy, runtime); the authors state that this paper does not imply that DNN watermarking is infeasible.

   Collapse this answer

Related papers