Detecting, Explaining, and Mitigating Memorization in Diffusion Models
Authors: Yuxin Wen, Yuchen Liu, Chen Chen, Lingjuan Lyu
ICLR 2024 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | In terms of efficacy, our method achieves an AUC of 0.960 and a TPR@1%FPR of 0.760 in under 2 seconds. In contrast, the baseline method (Carlini et al., 2023) demands over 39 seconds, even though it registers an AUC of 0.934 and a TPR@1%FPR of 0.523. To evaluate our detection method, we use 500 memorized prompts identified in Webster (2023) for Stable Diffusion v1 (Rombach et al., 2022). |
| Researcher Affiliation | Collaboration | Yuxin Wen1 , Yuchen Liu2 , Chen Chen3, Lingjuan Lyu3 1University of Maryland, 2Zhejiang University, 3Sony AI ywen@umd.edu, yuchen.liu.a@zju.edu.cn {Chen A.Chen,Lingjuan.Lv}@sony.com |
| Pseudocode | No | No pseudocode or algorithm blocks were found. |
| Open Source Code | Yes | Code is available at https://github.com/ Yuxin Wen Rick/diffusion_memorization. |
| Open Datasets | Yes | To evaluate our detection method, we use 500 memorized prompts identified in Webster (2023) for Stable Diffusion v1 (Rombach et al., 2022)... Additionally, we use another 2, 000 prompts, evenly distributed from sources LAION (Schuhmann et al., 2022), COCO (Lin et al., 2014), Lexica.art (Santana, 2022), and randomly generated strings. |
| Dataset Splits | No | The paper describes the data used for fine-tuning and evaluation, but does not explicitly provide percentages or sample counts for training, validation, and test splits for the models being trained or fine-tuned, nor does it specify validation for hyperparameter tuning beyond varying ltarget and τ for mitigation strategies. It defines a set of memorized and non-memorized prompts for evaluating the detection method, which serve as a test set, but not standard train/val splits. |
| Hardware Specification | Yes | Meanwhile, we report the running time in seconds with a batch size of 4 on a single NVIDIA RTX A6000. |
| Software Dependencies | No | The paper mentions specific models and optimizers like 'Stable Diffusion v1', 'DDIM', 'CLIP text encoder', and 'Adam optimizer (Kingma & Ba, 2014)', but does not provide specific version numbers for these or other software dependencies. |
| Experiment Setup | Yes | All generations employ DDIM (Song et al., 2020) with 50 inference steps. We use Adam optimizer (Kingma & Ba, 2014) with a learning rate of 0.05 and at most 10 steps. For performance metrics, we compute the SSCD similarity score (Pizzi et al., 2022; Somepalli et al., 2023b) to gauge the degree of memorization by comparing the generation to the original image. Additionally, the CLIP score (Radford et al., 2021) is used to quantify the alignment between the generation and its corresponding prompt. Our experiments encompass 5 distinct fine-tuned models, each embedded with different memorized prompts, and the results are averages over 5 runs with different random seeds. In our evaluation of the proposed method, we test 5 distinct target losses ltarget, ranging from 1 to 5, for inference-time mitigation. We use Adam optimizer (Kingma & Ba, 2014) with a learning rate of 0.05 and at most 10 steps. Simultaneously, we investigate 5 different thresholds τ, spanning from 2 to 6, for training-time mitigation. For comparison, we use the most effective method from (Somepalli et al., 2023b), random token addition (RTA), as the baseline, which inserts 1, 2, 4, 6, or 8 random tokens to the prompt. In Fig. 5, we display a selection of qualitative results, setting ltarget = 3 and τ = 4 for our method, while adding 4 random tokens as a baseline strategy. |