Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in [1].
Evaluating Durability: Benchmark Insights into Image and Text Watermarking
Authors: Jielin Qiu, William Han, Xuandong Zhao, Shangbang Long, Christos Faloutsos, Lei Li
DMLR 2024 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Our study evaluates watermark robustness in both image and text domains, testing against an extensive set of 100 image perturbations and 63 text perturbations. The results reveal significant vulnerabilities in contemporary watermarking approaches detection accuracy deteriorates by more than 50% under common perturbations, highlighting a critical gap between current capabilities and practical requirements. These findings emphasize the urgent need for more robust watermarking methods that can withstand real-world disturbances. Our project website can be found at https://mmwatermark-robustness.github.io/. Keywords: image and text watermarking, robustness, image corruptions and text perturbations, multimodal |
| Researcher Affiliation | Collaboration | Jielin Qiu1,2 EMAIL William Han2 EMAIL Xuandong Zhao3 EMAIL Shangbang Long1 EMAIL Christos Faloutsos2 EMAIL Lei Li2 EMAIL 1 Google Deep Mind 2 Carnegie Mellon University 3 University of California, Berkeley |
| Pseudocode | No | The paper describes methods and procedures narratively within the text and via citations to other works. There are no explicit blocks labeled 'Pseudocode' or 'Algorithm'. |
| Open Source Code | Yes | We have publicly released our codebase at https://mmwatermark-robustness.github. io/ with CC BY-NC-SA License. |
| Open Datasets | Yes | We utilized 5,000 image-caption pairs from the COCO validation split (Lin et al., 2014). |
| Dataset Splits | Yes | We utilized 5,000 image-caption pairs from the COCO validation split (Lin et al., 2014). |
| Hardware Specification | Yes | We used 16 NVIDIA A6000 GPUs for our experiments. |
| Software Dependencies | No | The paper discusses various watermarking methods and models, often referencing prior work or default parameters (e.g., 'Res Net-50 trained with DINO and PCA whitening', 'U-Net', 'spatial transformer network', 'Bi Se Net'). However, it does not explicitly list specific version numbers for general software dependencies like Python, PyTorch, or TensorFlow. |
| Experiment Setup | Yes | KGW-WM We utilize game and delta values of 0.25 and 2.0, respectively. We also set the seeding scheme as simple_1 , which represents a simple bigram hash, to utilize the main settings of the experiments in the paper (Kirchenbauer et al., 2023a). During detection, we also utilize a z threshold of 0.5 and ignore repeated n-grams. KTH-WM We set the desired length of the generated text, m, to 30 as detailed in Kuditipudi et al. (2023). The length of the watermark sequence, n, is kept at the standard value of 256. For generating the random watermark sequence, we employ a key of 42. The authors method of evaluating their watermarking framework involves p-values, and we consider texts with p < 0.1 as effectively watermarked. Blackbox-WM We employ the τ word value of 0.8, and a λ value of 0.83 (Yang et al., 2023). We also use the embed mode during watermarking. During detection, if the confidence value is over 80%, we deem the detection algorithm as successfully finding the watermark. Unigram-WM When applying the watermark, we utilize a fraction and strength value of 0.5 and 2.0, respectively. Additionally, we determined the watermark key to be defaulted to 0. During detection, we utilize the default value of 6.0 (Zhao et al., 2023a). |