Unbiased Watermark for Large Language Models
Authors: Zhengmian Hu, Lichang Chen, Xidong Wu, Yihan Wu, Hongyang Zhang, Heng Huang
ICLR 2024 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We propose two innovative and practical watermarking techniques known as δ-reweight and γreweight. Through extensive experimentation, we demonstrate that these techniques preserve output quality in machine translation and text summarization tasks. We evaluate the performance of our Unbiased Watermarks on two important applications of seq2seq models: text summarization (TS) and machine translation (MT). |
| Researcher Affiliation | Academia | 1Department of Computer Science, University of Maryland, College Park, MD 20742, USA 2Department of ECE, University of Pittsburgh, Pittsburgh, PA 15261, USA 3School of Computer Science, University of Waterloo, Waterloo, ON N2L 3G1, Canada |
| Pseudocode | Yes | Algorithm 1 Watermarking framework |
| Open Source Code | No | In our experimental implementation, we took into consideration the effective precision of computer floating-point numbers. To ensure numerical stability and prevent NaNs, we implemented the algorithm in log space. This makes the algorithm more complex, and additionally, we designed the algorithm with grid search by reusing previous computation results for acceleration. We also implemented such algorithm with tensor operator for efficient computation on GPU. For more details, please refer to the source code. |
| Open Datasets | Yes | For the TS task, we use the BARTlarge model (Liu et al., 2020) and the CNN-DM (Hermann et al., 2015) corpus as our testing dataset. The MT task involves translating English to Romanian, for which we employ the Multilingual BART (MBart) (Liu et al., 2020) model on the WMT 14 En-Ro corpus. |
| Dataset Splits | No | The paper mentions the CNN-DM corpus (11,490 examples) and the WMT 14 En-Ro corpus (test set size of 1,999 examples), but does not explicitly state the training, validation, or test splits for these datasets, only specifying the total size or test set size. |
| Hardware Specification | Yes | Our experiments are carried out on a machine equipped with 2x AMD EPYC 7513 32-Core Processor and 8x A6000 GPUs. |
| Software Dependencies | No | The experiments are implemented based on the Huggingface library (Wolf et al., 2019), a popular platform for developing and sharing models in the NLP community. |
| Experiment Setup | Yes | We test δ with three values: 0.0, 1.0, 2.0, and fix γ to be 1/2. It is important to clarify that the δ and γ in our δ-reweight and γ-reweight are different from those in Kirchenbauer et al.’s method. In the latter, δ and γ are hyperparameters, while in our method, δ-reweight and γ-reweight are names of two reweighting strategies. ... For context code generation, we employ the most recent five tokens as context code. ... For the unbiased watermark code generation function, we use SHA-256 as the hash function and a 1024-bit random bitstring as the key k. The watermark code E is sampled from PE using hash(c, k) as the random seed. |