What makes unlearning hard and what to do about it
Authors: KAIRAN ZHAO, Meghdad Kurmanji, George-Octavian Bărbulescu, Eleni Triantafillou, Peter Triantafillou
NeurIPS 2024 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Our evaluation on forget sets that isolate these identified factors reveals previously-unknown behaviours of state-of-the-art algorithms that don t materialize on random forget sets. Based on our insights, we develop a framework coined Refined-Unlearning Meta-algorithm (RUM) that encompasses: (i) refining the forget set into homogenized subsets, according to different characteristics; and (ii) a meta-algorithm that employs existing algorithms to unlearn each subset and finally delivers a model that has unlearned the overall forget set. RUM substantially improves top-performing unlearning algorithms. Overall, we view our work as an important step in deepening our scientific understanding of unlearning and revealing new pathways to improving the state-of-the-art. |
| Researcher Affiliation | Collaboration | Kairan Zhao University of Warwick Meghdad Kurmanji University of Cambridge George-Octavian Barbulescu University of Warwick Eleni Triantafillou Google Deep Mind Peter Triantafillou University of Warwick |
| Pseudocode | No | The paper describes algorithms and frameworks but does not include structured pseudocode or algorithm blocks. |
| Open Source Code | Yes | Code is available at: https://github.com/kairanzhao/RUM |
| Open Datasets | Yes | We use the CIFAR-10, CIFAR-100, and Tiny-Image Net datasets for our evaluation. |
| Dataset Splits | Yes | For CIFAR-10 and CIFAR-100, the train, validation, and test set sizes are 45,000, 5,000, and 10,000 images, respectively. |
| Hardware Specification | Yes | All training was conducted on Nvidia RTX A5000 GPUs. |
| Software Dependencies | No | The paper mentions SGD optimizer, ResNet, VGG16, but does not provide specific version numbers for software dependencies or libraries. |
| Experiment Setup | Yes | For CIFAR-10, the Res Net-18 model was trained for 30 epochs using the SGD optimizer. The learning rate was initialized at 0.1 and scheduled with cosine decay. For CIFAR-100, the Res Net-50 model was trained for 150 epochs using the SGD optimizer, with the learning rate initialized at 0.1 and decayed by a factor of 0.2 at epochs 60 and 120. For Tiny-Image Net, the Res Net-18 model was trained for 80 epochs, and the VGG-16 model was trained for 100 epochs, both using the SGD optimizer with an initial learning rate of 0.1 and cosine decay. All models were trained with a weight decay of 0.0005, a momentum of 0.9, and a batch size of 256. |