Disguised Copyright Infringement of Latent Diffusion Models
Authors: Yiwei Lu, Matthew Y. R. Yang, Zuoqiu Liu, Gautam Kamath, Yaoliang Yu
ICML 2024 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Our results show that it is possible to conceal copyrighted images within the training dataset for LDMs. The encoder architecture in LDMs is part of an autoencoder architecture (e.g., the KL-based VAE), where the encoder and decoder are used separately for encoding and inference. Consequently, the decoder D can be naturally used to detect disguises qualitatively. Specifically, for a well-trained autoencoder, D(E(xc)) xc, while for disguises E(xd) E(xc), thus we have D(E(xd)) xc. In Figure 7, we show that the encoder-decoder architecture is a great detection tool for disguises, where the output of the autoencoder reveals the copyrighted content hidden in xd. |
| Researcher Affiliation | Academia | 1School of Computer Science, University of Waterloo, Waterloo, Canada 2Vector Institute. |
| Pseudocode | Yes | Algorithm 1 Disguise Generation Input: copyrighted image xc, base image xb, pre-trained encoder E, input threshold γ1, feature threshold γ2, distance measure on input space D1( ), distance measure on feature space D2( ), hyperparameter on input space constraint α, learning rate η. 1 Initialize disguise xd with base image xb 2 repeat 3 D1 D1(xb, xd) // image distance 4 D2 D2(E(xc), E(xd)) // feature distance 5 L αD1 + D2 // calculate loss 6 xd xd η L xd // update disguise 7 xd ProjΓ(xd) // project to admissible set 8 until D1 γ1 and D2 γ2 9 return disguise xd |
| Open Source Code | Yes | Our code is available at https://github.com/watml/d isguised_copyright_infringement. |
| Open Datasets | Yes | LDM: We adopt the official Py Torch implementation1 of conditional LDM (Rombach et al. 2022) and acquire the pre-trained weights2(including that of the encoder E, the denoising U-Net εθ and the text embedding cθ( )) of a 1.45B parameter KL-regularized LDM-8 (8 denotes downsampling factor) model conditioned on language prompts on LAION-400M (Schuhmann et al. 2021). Specifically, we randomly take a subset of 1000 images from Celeb A-HQ/25615 as clean training samples... 15https://www.tensorflow.org/datasets/catalog/celeb_a_hq |
| Dataset Splits | No | The paper describes the datasets used and the training/fine-tuning process but does not explicitly provide details about training, validation, and test splits with specific percentages, counts, or splitting methodologies. For example, for Textual Inversion, it mentions using 'a small set of images' but no explicit split. |
| Hardware Specification | No | The paper does not explicitly describe the specific hardware used to run its experiments, such as GPU or CPU models. |
| Software Dependencies | No | The paper mentions using 'official Py Torch implementation' for LDM and Textual Inversion, and 'BERT text encoder', but it does not provide specific version numbers for PyTorch or any other software dependencies required for replication. |
| Experiment Setup | Yes | Throughout our experiments, we apply the pre-trained KL-regularized encoder E, and set the input distance measure D1( ) as a sum of the multi-scale structural similarity index (MS-SSIM) loss (Wang et al. 2003) and L1 loss following the analysis of (Khare et al. 2021), and the feature distance measure to be the L2 loss: D2(E(xc), E(xd)) = E(xc) E(xd) 2. The choice of the copyrighted image xc, base image xb, the input threshold γ1, the feature threshold γ2 and the hyperparameter on the input constraint α are task-dependent (for different copyrighted material), and we specify them in their corresponding paragraphs below. We set the admissible set to be in the range of [0, 1] as the legitimate (normalized) image pixel value and run the algorithm for 100000 epochs (early stop if the stopping criteria are reached) for all experiments. For example, for Disguised symbol: We set γ1 = 0.05, γ2 = 0.35 (normalized to the range of [0, 1]) as the threshold of input and feature distance, respectively, and α = 8000. |