Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in [1].
Trustworthy Machine Learning through Data-Specific Indistinguishability
Authors: Hanshen Xiao, Zhen Yang, G. Edward Suh
ICML 2025 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | The experimental results on memorization mitigation, backdoor defense, and copyright protection show both the efficiency and effectiveness of the DSI noise mechanism. |
| Researcher Affiliation | Collaboration | Hanshen Xiao 1 2 Zhen Yang 2 G. Edward Suh 1 1NVIDIA 2Department of Computer Science, Purdue University. |
| Pseudocode | Yes | Algorithm 1 Optimal DSI Gaussian Mechanism Algorithm 2 DSI Deep Learning Framework |
| Open Source Code | No | The paper does not contain an explicit statement about the release of source code for the described methodology, nor does it provide a link to a code repository. |
| Open Datasets | Yes | We study the memorization when fine-tuning GPT2 (Radford et al., 2019) and Open Pre-trained Transformer (OPT) (Zhang et al., 2022) using Wiki Text-5 dataset (Merity et al., 2016). ... We evaluate the DSI defense on poisoned CIFAR-10 dataset |
| Dataset Splits | No | For each epoch, the CIFAR-10 training set of 50,000 samples, as our input set U, is partitioned into 125 batches, each containing 400 samples. ... The samples are then reformatted into 5,640 sequences, each of length 1,024, with each sequence treated as an individual sample. ... We observe that larger models, requiring more iterations (more exposure to the training data) before convergence, tend to memorize more information. For example, under the same duplication setting of v = 10, OPT-350M, which also exhibits lower perplexity, achieves a 39.2% (10, 100, 5) exact memorization rate, compared to 17.9% for OPT-125M, aligning with prior results in (Tirumala et al., 2022). Similarly, provable DSI guaran- tees provide conservative upper bounds. For instance, in the case of GPT2-small, we evaluate the same exact memorization on unseen test data, which averages 1.66% as our reference. |
| Hardware Specification | No | The paper does not specify the exact hardware (e.g., GPU/CPU models, memory) used for running the experiments. It mentions models like GPT2, OPT, ResNet-20, Wide ResNet-16, Pre Act-ResNet18, and Stable Diffusion, which are typically run on GPUs, but specific hardware details are not provided. |
| Software Dependencies | No | The paper mentions using 'standard Adam (Kingma, 2014)' as an optimizer but does not specify any software libraries (e.g., PyTorch, TensorFlow) or their version numbers. |
| Experiment Setup | Yes | For ϵ = 1, ϵ = 2, ϵ [3 : 4], and ϵ [5 : 8], we select the epoch number to be 3, 6, 10 and 15, respectively. We uniformly select the learning rate of the local SGD to be 1.25 10 4. ... For GPT-2 small, we use a learning rate of 3 10 4 and train for 1 epoch. For OPT-125M, we use a learning rate of 1.5 10 4 and train for 1 epoch. For OPT-350M, we use a learning rate of 1 10 5 and train for 5 epochs. ... For the hyper-parameter selection in our experiments shown in Fig. 3 and 4 we select T = 50 and K = 10 with a step size (learning rate) 10 3. |