Measuring Forgetting of Memorized Training Examples
Authors: Matthew Jagielski, Om Thakkar, Florian Tramer, Daphne Ippolito, Katherine Lee, Nicholas Carlini, Eric Wallace, Shuang Song, Abhradeep Guha Thakurta, Nicolas Papernot, Chiyuan Zhang
ICLR 2023 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We show that, while non-convex models can memorize data forever in the worst-case, standard image, speech, and language models empirically do forget examples over time.We design a methodology for measuring whether, and how quickly, individual examples are forgotten and become less vulnerable to privacy attacks. Our methodology builds on state-of-the-art membership inference attacks (Carlini et al., 2021a; Ye et al., 2021), the best known method for testing whether a given point was used in training. We use our methodology to show that, for deep neural networks trained on language, speech, or vision tasks, examples used early in training (and not repeated later on) are indeed forgotten by the model. |
| Researcher Affiliation | Collaboration | 1Google, 2ETH Zurich, 3Carnegie Mellon University, 4Cornell University 5University of California, Berkeley, 6University of Toronto |
| Pseudocode | Yes | A ALGORITHMS FOR MEASURING FORGETTING We present the algorithm for POISON in Algorithm 1 and the algorithm for INJECT in Algorithm 2. Algorithm 1: Monitoring Forgetting Throughout Training with POISON and Algorithm 2: Monitoring Forgetting Throughout Training with INJECT and Algorithm 3: Mean Estimation with Deterministic Ordering and Algorithm 4: Mean Estimation with Random Sampling and Injection |
| Open Source Code | No | The paper does not contain an explicit statement or link indicating the release of source code for the methodology described. |
| Open Datasets | Yes | We investigate forgetting across a variety of datasets and models. We use canary extraction to evaluate generative models (i.e., for Libri Speech and LMs) and MI for classification models (i.e., for Image Net).We train decoder-only, 110M parameter, Transformer-based language models (with the T5 codebase and architecture from Raffel et al. (2020)) for one epoch over a version of C4 (Dodge et al. (2021)) that had been deduplicated with Min Hash (Lee et al. (2021)). |
| Dataset Splits | No | We train Res Net-50 models for 90 epochs (each epoch is roughly 5,000 steps). |
| Hardware Specification | No | The paper describes the models and architectures (e.g., Res Net-50, Conformer (L), Transformer-based language models) but does not specify the hardware (e.g., GPU models, CPU types) used for running the experiments. |
| Software Dependencies | No | We train each model with a batch size of 2,048 utterances for 100,000 steps using the Adam optimizer (Kingma & Ba, 2015) and a transformer learning rate schedule (Vaswani et al., 2017).We train decoder-only, 110M parameter, language models (with the T5 codebase and architecture from Raffel et al. (2020)). |
| Experiment Setup | Yes | Image Net. We train Res Net-50 models for 90 epochs, with a learning rate of 0.1, momentum of 0.9, and batch size of 256 (making each epoch roughly 5,000 steps). and Libri Speech. We train each model with a batch size of 2,048 utterances for 100,000 steps using the Adam optimizer (Kingma & Ba, 2015) and a transformer learning rate schedule (Vaswani et al., 2017). |