Membership Inference Attacks against Large Vision-Language Models
Authors: Zhan Li, Yongtao Wu, Yihang Chen, Francesco Tonin, Elias Abad Rocamora, Volkan Cevher
NeurIPS 2024 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | In this study, we introduce the first membership inference attack (MIA) benchmark tailored for various VLLMs to facilitate training data detection. Then, we propose a novel MIA pipeline specifically designed for token-level image detection. Lastly, we present a new metric called Max Rényi-K%, which is based on the confidence of the model output and applies to both text and image data. We believe that our work can deepen the understanding and methodology of MIAs in the context of VLLMs. Our code and datasets are available at https://github.com/LIONS-EPFL/VL-MIA. |
| Researcher Affiliation | Academia | Zhan Li Yongtao Wu Yihang Chen Francesco Tonin Elias Abad Rocamora Volkan Cevher LIONS, EPFL [first name].[last name]@epfl.ch |
| Pseudocode | No | The paper includes schematic diagrams (e.g., Figure 1) but not structured pseudocode or algorithm blocks. |
| Open Source Code | Yes | Our code and datasets are available at https://github.com/LIONS-EPFL/VL-MIA. |
| Open Datasets | Yes | We construct a general dataset: Vision Language MIA (VL-MIA), based on the training data used for popular VLLMs, which, to our knowledge, is the first MIA dataset designed specifically for VLLMs. We present a takeaway overview of VL-MIA in Table 1. We also provide some examples in VL-MIA, see Table 16 in the appendix. The prompts we use for generation can be found in Table 6. [...] VL-MIA/Flickr. MS COCO [34] co-occurs as a widely used dataset in the training data of the target models, so we use the images in this dataset as member data. |
| Dataset Splits | No | The paper does not explicitly describe a separate validation set or its splits for training the MIA model. It defines member and non-member data for evaluation. |
| Hardware Specification | Yes | We run experiments on a single NVIDIA A100 80GB GPU, where the image MIA costs less than 2 hours for one model. |
| Software Dependencies | Yes | Table 7: Model details used in this work. Model Mini-GPT4 LLa VA 1.5 LLa MA Adapter v2.1 |
| Experiment Setup | Yes | In our experiments, we vary α = 1/2, 1, 2, and +∞; K = 0, 10, 100. As α increases, the top percentile of distribution p will have more influence on Hα(p). When K = 0, we define the Max Rényi-K% score to be maxi [L 1] Hα(p(i)). When K = 100, the Max Rényi-K% score is the averaged Rényi entropy of the sequence X. |