Algorithms that Approximate Data Removal: New Results and Limitations
Authors: Vinith Suriyakumar, Ashia C. Wilson
NeurIPS 2022 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Across a variety of benchmark datasets, our algorithm empirically improves upon the runtime of prior methods while maintaining the same memory requirements and test accuracy. Finally, we open a new direction of inquiry by proving that all approximate unlearning algorithms introduced so far fail to unlearn in problem settings where common hyperparameter tuning methods, such as cross-validation, have been used to select models. Experiments In this section we will refer to retraining from scratch as RT, the Algorithm (8) as TA and our Algorithm 1 as IJ. We empirically demonstrate the benefits of our algorithm over both RT and TA in three different settings: (i) smooth regularizers where we train a logistic regression model with an ℓ2 penalty to predict between the digit 3 and 8 from the MNIST dataset [19], (ii) non-smooth regularizers where we train a logistic regression model with an ℓ1 penalty to predict whether an individual was prescribed a Warfarin dosage of > 30 mg/week from a dataset released by the International Warfarin Pharmacogenetics Consortium [8], and (iii) non-convex training where we apply a logistic regression model with an ℓ2 penalty to predict street digits signs 3 and 8 from SVHN [24] on representations extracted from a differentially private feature extractor with ϵ = 0.1 (similar to the setup of Guo et al. [14]). |
| Researcher Affiliation | Academia | Vinith M. Suriyakumar MIT EECS vinithms@mit.edu Ashia C. Wilson MIT EECS ashia07@mit.edu |
| Pseudocode | Yes | Algorithm 1 Infinitesimal Jacknife (IJ) Online Unlearning Algorithm |
| Open Source Code | Yes | In the Appendix D, we provide further details about each dataset and the code to produce our models is attached separately. (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)? [Yes] |
| Open Datasets | Yes | MNIST dataset [19], International Warfarin Pharmacogenetics Consortium [8], SVHN [24] |
| Dataset Splits | No | The paper mentions training, testing, and tuning hyperparameters but does not provide explicit training/validation/test dataset splits (e.g., 80/10/10 split or specific sample counts for each split). |
| Hardware Specification | Yes | All models are trained on a single NVIDIA Tesla T4 GPU and 16 2.10GHz Xeon(R) Silver 4110 CPU cores. |
| Software Dependencies | No | The paper does not specify version numbers for any software dependencies or libraries used for the experiments (e.g., PyTorch, TensorFlow, Scikit-learn versions). |
| Experiment Setup | Yes | For all algorithms, we tune λ over the set {10 3, 10 4, 10 5, 10 6}. In these experiments we consider noise at c = 0.01. |