Entangled Watermarks as a Defense against Model Extraction

1. 01What problem does this paper attempt to solve?

   Existing DNN watermarks are separate from the task distribution and are easily removed by model extraction; the paper proposes the entangled watermark EWE as a defense.

   Core question: how to make DNN watermarks survive model extraction, so that the defender can claim ownership of a stolen model.

   • Setting and motivation: training ML models is costly, and deployed models are valuable intellectual property; an attacker can steal one by querying the victim model and training a substitute model on the returned labels. The authors argue that preventing extraction without sacrificing accuracy for benign users is difficult.
   • Shortcomings of existing methods: watermarking makes the model overfit to outlier input-output pairs known only to the defender, but the authors point out that this is precisely its weakness—the watermark is an outlier outside the task distribution, the attacker queries only with task-distribution data, and the extracted model retains only the task-relevant decision surface, so the watermark is lost along with it (§3 illustrates this with an analytical example and neuron activation experiments).
   • Core observation: the authors claim that a watermarked model roughly splits its parameters into two sub-models, one encoding the task distribution and one overfitting the watermark; Figure 2 shows that in baseline watermarked models, legitimate data and watermark data activate different and fewer neurons.
   • Proposed method: the paper proposes Entangled Watermark Embedding (EWE), which uses a soft nearest neighbor loss to make the model recognize both task data and the watermark with the same set of parameters, so that removing the watermark necessarily harms the task accuracy of the extracted model.
   • Threat model (start of §4):
     • Goal: extract a copy of the model that does not contain the watermark.
     • Knowledge: knows the victim model's training data (but not the labels), knows the model architecture, knows that a watermark is deployed.
     • Capability: performs extraction using data points from the task distribution (retraining-based extraction).
     • Limitations: does not know the watermark calibration parameters (κ, temperature) or the trigger; the authors call this a powerful white-box attacker and state that this assumption is consistent with prior work.

   Collapse this answer

2. 02What related research exists?

   Related work is grouped into three parts: model extraction attacks, defenses against extraction, and ML watermarking; the baseline is the cross-entropy watermark of Adi et al.

   Model extraction attacks

   • Tramèr et al. [51] proposed stealing models through a prediction API and gave three uniform sampling techniques; Papernot et al. [43] trained substitute models with Jacobian-based data augmentation. The paper focuses on this type of attack, which "steals a copy with similar accuracy using only query predictions."
   • The paper explicitly excludes two types: differential query methods that require a large number of queries to obtain a functionally equivalent model [19, 35], and attacks that rely on side-channel information [3].

   Defenses against extraction

   • One class of countermeasures limits or modifies the information returned per query, such as reducing the precision of probability vectors (quantization), returning only the most likely label, returning random labels, or adding noise [19, 51, 29]. The paper points out that these countermeasures all involve an inherent trade-off between the model's utility for benign users and its resistance to extraction.

   ML watermarking

   • Watermarking originates from intellectual property protection for media [22]; Zhang et al. [56] and Nagai et al. [38] proposed watermarking algorithms based on data poisoning; Rouhani et al. [10] (DeepSigns) embed bits in the probability density functions of each layer. The paper points out that these methods share the common trait of treating the watermark as an outlier outside the task distribution, and thus it can be removed by extraction—this is the fundamental limitation the paper names.
   • Watermarking can be viewed as "backdoor injection for good" [15, 1]; Adi et al. [1] watermark DNNs via backdoors.

   Baselines and benchmarks: the baseline for the main results (Table 1) is the Adi et al. [1]-style watermark—using only cross-entropy to make the watermark classified into the target class (i.e., κ = 0); the backdoor defenses compared against in the defense evaluation are pruning, fine-pruning [31], and Neural Cleanse [53]. The experimental benchmarks are MNIST, Fashion MNIST, CIFAR-10, CIFAR-100, and Google Speech Commands.

   Positioning of this paper: the authors state that prior watermarking learns the watermark task and the main task separately, whereas EWE learns the two jointly, so that the watermark behavior is retained after extraction.

   Collapse this answer

3. 03How does the paper address this problem?

   EWE adds the SNNL of each layer to the training loss, forcing the watermark to share representations with the target task classes, so that extraction cannot strip the watermark.

   Overall idea: since the root cause of watermark removal is that the model learns the watermark and the task as two independent sub-models, make the two entangled in representation space—recognized by the same set of neurons. The method is named Entangled Watermark Embedding (EWE), see Algorithm 1.

   Core tool: Soft Nearest Neighbor Loss

   • Uses the SNNL from Frosst et al. [12] to measure the degree of entanglement of data manifolds of different groups in representation space: SNNL(X,Y,T) = −1/N∑_i log (∑_{j: yi=yj} e^{−||x_i−x_j||2/T})/(∑_{k≠ i} e^{−||x_i−x_k||2/T}), i.e., the ratio of the average distance among same-class points to the average distance among all points, with temperature T controlling the distance scale.
   • Minimizing SNNL separates classes into clusters, while maximizing SNNL makes clusters overlap; EWE maximizes entanglement between watermark and target-class data (taking a negative sign in the total loss).

   Step 1: Generate the watermark

   • The defender selects a watermark distribution Dw and a source class cS, samples Xw, and assigns labels of a semantically different target class cT; Dw can be the same as the task dataset (in-distribution) or come from a related dataset (OOD watermark).
   • The trigger location is taken as the region where the SNNL gradient with respect to the candidate input is largest (convolution in line 2 of Algorithm 1), to avoid the trigger changing the semantics of Xw so that it approaches cT.
   • Optional step: use FGSM to perform gradient ascent on the watermark (alternately optimizing cross-entropy and SNNL) to reduce the model's confidence in the target class; the authors say FGSM was chosen because a single-step large perturbation transfers more easily across models than PGD, and this paper does not care about watermark imperceptibility.

   Steps 2–3: Modify the loss and train

   • The total loss is L = L_CE(X,Y) − κ · ∑_l=1^L SNNL([X_w^(l), X_cT^(l)], Y′, T^(l)), i.e., entangling the watermark and target-class data at each layer's representation, with κ controlling the trade-off between watermark robustness and task accuracy.
   • During training, 1 mixed batch [Xw, XcT] is inserted every r normal batches; normal batches use κ = 0 and optimize only cross-entropy, while mixed batches use κ > 0. The temperature T is automatically fine-tuned during training according to the learning rate α, requiring no manual tuning.

   Ownership verification

   • The defender queries the suspected stolen model with watermark samples and uses a T-test to determine whether the watermark success rate exceeds the false alarm rate (set to random guessing 100/K%, which the authors call a conservative upper bound).
   • By the central limit theorem (the success rate is approximately normal when n > 30), Figure 4 gives the number of queries needed for 95% confidence: when the watermark success rate is above 23%, only about 30 queries are needed; the authors say the lowest success rate observed for EWE, 18.74% (CIFAR-10), also requires fewer than 100 queries.

   Collapse this answer

4. 04What experiments did the paper conduct?

   Comparing EWE with baselines on 5 datasets and 3 architectures: after extraction, EWE retains on average 38.39% of the watermark, versus 5.77% for the baseline.

   Experimental setup

   • Architectures evaluated: CNN (2 convolutional + 2 fully connected layers, for MNIST/Fashion MNIST), RNN (80 LSTM units, for Speech Commands), ResNet-18 (for Fashion MNIST/CIFAR-10/CIFAR-100).
   • Datasets: MNIST (70,000 images), Fashion MNIST, Google Speech Commands (about 40,000 training samples, converted to Mel spectrograms), CIFAR-10 (60,000 images), CIFAR-100; OOD watermarks are taken from Fashion MNIST/MNIST/SVHN.
   • Baseline: Adi et al. [1]-style watermark, using only cross-entropy to classify the watermark into the target class (κ = 0).
   • Metrics: watermark success rate (the proportion of watermarks classified as cT, reported in the main text after subtracting the false alarm rate) and validation accuracy; the values in Table 1 include standard deviations (the number of repetitions is not explicitly stated in the paper).
   • Extraction method: the victim model labels its own training data, and a copy is retrained (the white-box attacker uses only cross-entropy and retrains only on legitimate data).
   • Ownership determination: T-test, 95% confidence, with the false alarm rate set to random guessing (§4.3.1).

   Main results

   Table 1: validation accuracy / watermark success rate (%) of the victim model and the extracted model.

   Swipe horizontally to read the table

   Dataset	Method	Victim accuracy	Victim watermark success rate	Extracted accuracy	Extracted watermark success rate
   MNIST	Baseline / EWE	99.03 / 98.91	99.98 / 99.9	98.79 / 98.76	0.31 / 65.68
   Fashion MNIST	Baseline / EWE	90.48 / 90.31	98.76 / 87.83	89.8 / 89.82	8.96 / 58.1
   Speech Command	Baseline / EWE	98.11 / 97.5	98.67 / 96.49	97.3 / 96.83	3.55 / 41.65
   Fashion MNIST (ResNet)	Baseline / EWE	91.64 / 88.33	75.6 / 94.24	91.05 / 88.27	5.68 / 24.63
   CIFAR10	Baseline / EWE	85.82 / 85.41	19.9 / 25.74	81.62 / 81.78	7.83 / 18.74
   CIFAR100	Baseline / EWE	54.11 / 53.85	8.37 / 67.87	47.42 / 47.62	8.31 / 21.55

   • Authors' summary: after extraction, the EWE watermark success rate is 18%–60%, averaging 38.39%, versus 0.3%–9%, averaging 5.77%, for the baseline; validation accuracy drops by only 0.81 percentage points on average, with a maximum of 3 percentage points (ResNet on Fashion MNIST).
   • The authors state that this success rate is sufficient to claim ownership with fewer than 100 queries at 95% confidence.
   • Exceptions: on CIFAR-100 the watermark success rates of both the baseline and EWE are low (the authors attribute this to the large number of classes and the complex representation space); entanglement is harder on ResNet, and the authors state that residual connections favor linearity and that the watermark shares no features with legitimate data, so they switch to an OOD watermark.

   Verification of entanglement degree (§4.3.2)

   • Representation visualization: Figure 3 projects the penultimate-layer representations with PCA; the baseline pushes the watermark into a separate cluster, while EWE makes the watermark overlap with the data cluster for cT = 7 (per the authors).
   • Activation patterns: Figure 2 shows that in the EWE model the activation patterns of the watermark and legitimate data are more similar; Figure 20 (Appendix A.3) shows that for the CNN, representations in deeper layers are more similar.
   • CKA similarity: Figure 5 shows that the larger the SNNL penalty weight κ, the higher the CKA similarity between watermark and legitimate-data representations (relative to the κ = 0 baseline).

   Resistance to backdoor defenses (§5.3)

   • Pruning: Figure 8 shows that to push the watermark success rate of the extracted model below 20%, task accuracy must drop by more than 40 percentage points; the authors state that at this point the model is no longer valuable to the attacker.
   • Fine-pruning: Figure 9 shows that under the most favorable setting the extracted model's watermark success rate is still around 20% (with 50% of neurons pruned), which the authors state is still sufficient to claim ownership.
   • Neural Cleanse: on Fashion MNIST, EWE has an average anomaly index of 1.24 (5 runs), below the detection threshold of 2; the baseline is 8.84 and would be detected. The UMAP projection in Figure 11 shows that the baseline brings the clusters of class 8 and class 0 closer together, whereas EWE does not noticeably change inter-cluster distances (per the authors).

   Adaptive attacks (§6)

   • Disentangling: the attacker knows all EWE parameters and only guesses the source-target class pair; with κ < 0 提取。Fashion MNIST 上水印成功率从 48.81% 降到 22.82%（猜错）或 6.34%（猜对）；MNIST 上从 41.62% 降到 30.14%（猜错）或 0.08%（猜对）；Speech Commands 平均降到 16.81%。作者称猜错时仍可用约 30 次（视觉）或近 100 次（语音）查询主张所有权，猜对需训练 K(K−1) 个模型、违背窃取目的。
   • Piracy 攻击：攻击者用 EWE 嵌入自己的水印造成归属歧义；作者用 fine-pruning 应对，Figure 12 显示剪枝不到 10% 神经元可去除盗版水印而所有者水印保留。
   • 异常检测：Table 2 显示 LOF 检出 99.93% 水印、准确率损失 7.00 个百分点；Isolation Forest 检出 92.82%、损失 8.64 个百分点（Fashion MNIST，最后隐藏层激活）。
   • 迁移学习：GTSRB → LISA，Figure 13 显示只微调全连接层准确率最高 98.25%、水印不受影响；微调全部层达 98.56% 并随学习率增大开始削弱水印，但作者称水印被去除前预训练知识已丢失。

   其他消融与分析

   • 温度 T（Figure 14）：对鲁棒性影响不大，作者归因于训练中自动微调。
   • 权重 κ（Figure 15）：κ 越大水印越鲁棒、任务准确率越低。
   • 比例 r（Figure 16）：Fashion MNIST 上 r > 3 the watermark can be extracted and removed, and when r < 1 task accuracy drops.
   • Source-target class pairs (Figures 17, 25): on Fashion MNIST, the higher the cosine similarity of class centers, the more robust the watermark; on Speech Commands the trend is not obvious.
   • Depth (Figure 19): a plain CNN with additional convolutional layers can still be entangled; Figure 24 shows that on CIFAR-100, as the number of classes increases, the accuracy of both the baseline and EWE is lower.

   Collapse this answer

5. 05What are some points that can be further explored?

   The authors list limitations and future directions including scalability, computational overhead, and layer selection; experiments cover 5 datasets and 3 architectures.

   Limitations and future directions noted by the authors

   • Parameter secrecy is a precondition (§6.1): If the attacker knows EWE's trigger and parameters, the watermark can be completely removed (reject classification, minimize SNNL to disentangle, retrain the trigger); the authors say this is outside their threat model, and the parameters act like a key.
   • The trade-off worsens when extending to deeper architectures and more complex datasets (§7 Scalability, §4.3.4): The CIFAR-100 results suggest that extending to larger datasets may require more work; the number of classes affects EWE's performance, which the authors suspect is due to a more complex representation space.
   • Computational overhead (§7): The watermark dataset should be 2 times smaller than the legitimate dataset, meaning training data becomes 1.5–2 times the original; the authors say a more detailed analysis is needed to confirm whether this phenomenon exists in more complex tasks.
   • Utility improvements (§7): Some applications do not accept even a small utility drop; the authors believe that replacing ReLU with a smoother Sigmoid when computing SNNL may make it easier to achieve the same properties.
   • Layer selection (§7): Figure 20 shows that adding SNNL only at the last few layers can achieve the required guarantee with slightly improved utility; how to select layers is left for future work.
   • Future directions (§7): Study the influence of trigger design and representation-space structure on source-target class selection; extend from 1-to-1 to m-to-n watermarks; determine the relationship between hyperparameters and dataset properties (§7 Hyperparameter Selection). The conclusion (§8) states that "extending EWE to complex tasks without greatly hurting accuracy remains an open problem."

   Scope of experiments

   • The architectures tested are CNN, RNN (LSTM), and ResNet-18, 3 in total; the datasets are MNIST, Fashion MNIST, CIFAR-10, CIFAR-100, and Google Speech Commands, 5 in total, plus a GTSRB→LISA transfer experiment.
   • The defense evaluation covers three backdoor defenses—pruning, fine-pruning, and Neural Cleanse—and four adaptive attacks—disentangling, piracy, anomaly detection (LOF, Isolation Forest), and transfer learning.
   • The extraction attack is retraining-based (retraining on training data labeled by the victim model); the paper explicitly excludes high-fidelity extraction with differential queries and side-channel attacks (§2.2).
   • Ownership verification uses a T-test at 95% confidence, with the false alarm rate set to random guessing; the Neural Cleanse anomaly index is the mean of 5 runs.
   • The paper does not report the number of repetitions for each value in Table 1, nor does it report the specific query budget used for extraction.

   Collapse this answer

6. 06Summarize the main content of the paper

   By entangling the watermark with task representations, EWE prevents model extraction from stripping the watermark, retaining on average 38.39% of the watermark with only a 0.81 percentage point drop in accuracy.

   A DNN watermarking defense paper against model stealing (USENIX Security 2021).

   • Problem: Existing watermarks are outliers outside the task distribution, and an attacker can remove the watermark simply by extracting the model using only task-distribution data; the authors use an analytical example and neuron activation experiments to show that a watermarked model is in fact two independent sub-models.
   • Method: Entangled Watermark Embedding (EWE) adds a Soft Nearest Neighbor Loss at each layer to the training loss, forcing watermark data and target-class data to share representations; trigger locations are selected by SNNL gradient, with optional FGSM optimization of the watermark; temperature T is learned automatically during training, and weight κ controls the robustness–accuracy trade-off.
   • Main results (Table 1): On MNIST, Fashion MNIST, Speech Commands, and CIFAR-10/100, after retraining-based extraction the EWE watermark success rate is 18.74%–65.68%, averaging 38.39%, versus only 0.31%–8.96% for the baseline, averaging 5.77%; validation accuracy loss averages 0.81 percentage points, with a maximum of 3 percentage points. The authors say this is enough to claim ownership with fewer than 100 queries at 95% confidence.
   • Robustness: pruning must sacrifice over 40 percentage points of accuracy to push the watermark below 20%; the Neural Cleanse anomaly index is 1.24, below the threshold of 2 (baseline 8.84); against four adaptive attacks—disentangling, piracy, anomaly detection, and transfer learning—the authors say the attacker likewise faces "no free lunch"—removing the watermark requires sacrificing the utility of the stolen model.
   • Exceptions: On CIFAR-100, both the baseline and EWE have low watermark success rates; entanglement is harder on ResNet, requiring a switch to OOD watermarks.
   • Authors' conclusion: EWE prevents a thief from removing the watermark without harming the accuracy of the copy; extending to more complex tasks without greatly hurting accuracy remains an open problem.

   Collapse this answer

Related papers