On the Difficulty of Defending Self-Supervised Learning against Model Extraction
Authors: Adam Dziedzic, Nikita Dhawan, Muhammad Ahmad Kaleem, Jonas Guan, Nicolas Papernot
ICML 2022 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We evaluate our attacks against ResNet-50 and ResNet-34 SimCLR victim models. We train the models and generate queries for the extraction attacks using ImageNet and CIFAR10, and evaluate the models downstream performance on CIFAR100, STL10, SVHN, and Fashion-MNIST. Our experimental results in Section 4 show that our attacks can steal a copy of the victim model that achieves considerable downstream performance in fewer than 1/5 of the queries used to train the victim. |
| Researcher Affiliation | Academia | Adam Dziedzic 1 2 Nikita Dhawan 1 2 Muhammad Ahmad Kaleem 1 2 Jonas Guan 1 2 Nicolas Papernot 1 2 1University of Toronto 2Vector Institute. |
| Pseudocode | Yes | Algorithm 1 Stealing an Encoder. Input: Querying Dataset D, access to a victim model fv(w; θv). Output: Stolen representation model fa(w; θa) |
| Open Source Code | No | The paper does not provide an explicit statement about releasing its own source code or a link to a code repository. |
| Open Datasets | Yes | We include results for victim models trained on the Image Net, CIFAR10, and SVHN datatsets. The ImageNet encoder has an output representation dimension of 2048, while encoders trained on CIFAR10 and SVHN return 512 dimensional representations. For ImageNet, we use the publicly available ResNet50 model from (Chen & He, 2020). |
| Dataset Splits | No | The paper mentions using training and test sets but does not explicitly provide details about a validation dataset split or how it was used for hyperparameter tuning or early stopping. |
| Hardware Specification | Yes | We ran all experiments on machines equipped with an Intel Xeon Silver 4210 processor, 128 GB of RAM, and four NVIDIA GeForce RTX 2080 graphics cards, running Ubuntu 18.04. |
| Software Dependencies | No | The paper mentions 'PyTorch' and 'Ubuntu 18.04' but does not provide specific version numbers for libraries or frameworks crucial for reproducibility beyond the operating system. |
| Experiment Setup | Yes | For the CIFAR10 and SVHN datasets, we use a public PyTorch implementation of SimCLR (Chen et al., 2020) to train victim ResNet18 and ResNet34 models over 200 epochs with a batch size of 256 and a learning rate of 0.0003 with the Cosine Annealing Scheduler and the Adam optimizer. For training stolen models, we use similar (hyper-)parameters to the training of the victim models, with a batch size of either 64 or 256, initial learning rate of 0.0001, and the Adam optimizer. In the case of stealing from the ImageNet victim model, we use a larger learning rate of 0.1 or 1.0 with the LARS optimizer (Ginsburg et al., 2018) and a batch size of 256 or 512. |