Private Attribute Inference from Images with Vision-Language Models
Authors: Batuhan Tömekçe, Mark Vero, Robin Staab, Martin Vechev
NeurIPS 2024 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | On this dataset, we evaluate 7 state-of-the-art VLMs, finding that they can infer various personal attributes at up to 77.6% accuracy. |
| Researcher Affiliation | Academia | Batuhan Tömekçe, Mark Vero, Robin Staab, Martin Vechev Department of Computer Science ETH Zurich tbatuhan@ethz.ch {mark.vero,robin.staab,martin.vechev}@inf.ethz.ch |
| Pseudocode | No | The paper does not contain any structured pseudocode or algorithm blocks. |
| Open Source Code | Yes | An open-source implementation1 of our dataset labeling tool and our inference pipeline to advance privacy research. 1Code available at: https://github.com/eth-sri/privacy-inference-multimodal |
| Open Datasets | No | Due to the sensitive nature of such datasets and in line with previous works as well as ethical concerns, we decided not to release the VIP dataset publicly. |
| Dataset Splits | No | The paper describes the VIP dataset used for evaluation but does not specify any training, validation, or test splits for this dataset within the context of their experiments. All models are run for every image-attribute pair in the VIP dataset. |
| Hardware Specification | Yes | All open-source models were run on a single Nvidia-H100 GPU instance. |
| Software Dependencies | No | The paper mentions specific versions of proprietary models used (e.g., "gpt-4-1106vision-preview", "gemini-pro-vision") but does not provide version numbers for general ancillary software dependencies like programming languages or libraries (e.g., Python, PyTorch). |
| Experiment Setup | Yes | Unless mentioned explicitly, we use a single-round prompt with the models, not allowing for zooming, which we evaluate in a separate experiment. As described in Section 3, all proprietary models are aligned with safeguards. Therefore, we query these models via a gamified and Co T-extended prompt (later referred to as "Final" prompt) presented in Appendix E.3. We do so also for LLa Va-Ne XT 34B and Intern VL-Chat-V1.2-Plus. As Cog Agent-VQA, Idefics 80B, and LLa Va 1.5 13B exhibit weaker language understanding capabilities and are mostly free from safeguards, we evaluate them with a simpler prompt (presented in Appendix E.5). Our prompting choices are motivated by avoiding the underreporting of the model s inference capabilities, and as such, potentially downplaying the posed privacy risk. We ablate the specific choice of prompts for all open-source models in Appendix B.3. |