PID: Prompt-Independent Data Protection Against Latent Diffusion Models
Authors: Ang Li, Yichuan Mo, Mingjie Li, Yisen Wang
ICML 2024 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | In this paper, we first empirically demonstrate that breaking this assumption, i.e., in cases where discrepancies exist between the textual conditions used by protectors and exploiters, could substantially reduces the effectiveness of these defenses. Furthermore, considering the visual encoder s independence from textual prompts, we delve into the visual encoder and thoroughly investigate how manipulating the visual encoder affects the few-shot fine-tuning process of LDMs. Drawing on these insights, we propose a simple yet effective method called Prompt-Independent Defense (PID) to safeguard privacy against LDMs. We show that PID can act as a strong privacy shield on its own while requiring significantly less computational power. We believe our studies, along with the comprehensive understanding and new defense method, provide a notable advance toward reliable data protection against LDMs. Our code is available at https://github.com/PKUML/Diffusion-PID-Protection |
| Researcher Affiliation | Academia | 1School of EECS, Peking University, China 2National Key Lab of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University, China 3CISPA Helmholtz Center for Information Security, Germany 4Institute for Artificial Intelligence, Peking University, China. |
| Pseudocode | No | The paper does not contain structured pseudocode or algorithm blocks. It describes mathematical formulations and procedures in prose. |
| Open Source Code | Yes | Our code is available at https://github.com/PKUML/Diffusion-PID-Protection |
| Open Datasets | Yes | Our experiments primarily utilize the Celeb A-HQ (Liu et al., 2015) dataset where we randomly select 10 celebrities and choose 4 images for each. ... We compare PID with the three symbolic defense methods, Adv DM (Liang et al., 2023), FSGM, and ASPL (Van Le et al., 2023) on the Celeb A-HQ (Liu et al., 2015) and VGGFACE (Cao et al., 2018) dataset. |
| Dataset Splits | No | The paper specifies that 'we randomly select 10 celebrities and choose 4 images for each' for fine-tuning Latent Diffusion Models. While these 4 images act as the training data for fine-tuning, the paper does not specify traditional training/validation/test dataset splits for the overall dataset used (e.g., Celeb A-HQ) to facilitate typical model validation or hyperparameter tuning. The evaluation metrics are applied to *generated* images, not a separate validation split of the dataset. |
| Hardware Specification | No | The paper mentions 'consuming significantly less computational resources (approximately 20% GPU memory, 5G v.s. 24G)' when comparing PID, but it does not specify any particular GPU models (e.g., NVIDIA A100, RTX 3090), CPU models, or other specific hardware components used for the experiments. |
| Software Dependencies | No | The paper mentions using 'Stable Diffusion v1.5' and 'Stable Diffusion v2.1' as base models, and 'Dream Booth' and 'Lo RA' for fine-tuning. It also cites 'CLIP'. While these are software components, specific version numbers for underlying libraries like PyTorch, TensorFlow, or specific versions of HuggingFace Transformers are not provided. |
| Experiment Setup | Yes | The perturbation budget is set to 0.05 and the perturbed images are saved in PNG format in this paper unless otherwise specified. ... For Adv DM (Liang et al., 2023), we implement the protection loss as Lsemantic + λLtextural with λ = 0.05... We run the algorithm for 100 steps... with step size equaling ε/10. For FSGM, we run the defense for 100 iterations, with step size being ε/10 and 1-step gradient accumulation. For ASPL, we run the defense for 50 iterations... Table 8. Fine-tuning hyper-parameters. Version Freeze-TE LR Steps Batch Size Grad. Accu. Output Res. 1.5 Yes 2e-6 1000 1 1 512x512 |