Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in [1].
Understanding Fine-tuning in Approximate Unlearning: A Theoretical Perspective
Authors: Meng Ding, Rohan Sharma, Changyou Chen, Jinhui Xu, Kaiyi Ji
TMLR 2025 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Our theoretical analysis reveals that while FT models can achieve zero remaining loss, they fail to forget the forgetting data, as the pretrained model retains its influence and the fine-tuning process does not adequately mitigate it. To address this, we propose a novel Retention-Based Masking (RBM) strategy that constructs a weight saliency map based on the remaining dataset, unlike existing methods that focus on the forgetting dataset. Our theoretical analysis demonstrates that RBM not only significantly improves unlearning accuracy (UA) but also ensures higher retaining accuracy (RA) by preserving overlapping features shared between the forgetting and remaining datasets. Experiments on synthetic and real-world datasets validate our theoretical insights, showing that RBM outperforms existing masking approaches in balancing UA, RA, and disparity metrics. |
| Researcher Affiliation | Academia | Meng Ding Department of Computer Science and Engineering State University of New York at Buffalo Rohan Sharma Department of Computer Science and Engineering State University of New York at Buffalo Changyou Chen Department of Computer Science and Engineering State University of New York at Buffalo Jinhui Xu School of Information Science and Technology University of Science and Technology of China Kaiyi Ji Department of Computer Science and Engineering State University of New York at Buffalo |
| Pseudocode | No | The paper describes methods using mathematical formulations and descriptive text, but does not include any explicit pseudocode or algorithm blocks. For example, the Retention-Based Masking strategy is described with equations (9) and (10) and accompanying text. |
| Open Source Code | No | The paper does not provide any explicit statements about releasing source code, nor does it include links to code repositories. It mentions 'following the recommendations from the official repository' for 'Sal Un', which refers to a third-party implementation, not the authors' own work described in the paper. |
| Open Datasets | Yes | Experiments on synthetic and real-world datasets validate our theoretical insights... We conduct our evaluations using the Res Net-18 backbone across the methods... Our experiments will focus on image classification using the CIFAR-10 (Krizhevsky et al., 2009) and CIFAR-100 (Krizhevsky et al., 2009)... We further evaluate our approach on the Tiny Image Net dataset... we adopt the Vi T-Tiny model... and evaluate it on the SVHN dataset. |
| Dataset Splits | Yes | This dataset [Tiny Image Net] is a subset of Image Net, containing 200 classes with 500 training samples per class and 50 samples each for validation and testing. We consider a random forgetting scenario involving 10% of the data, with a masking ratio of 50% for both methods... We randomly forget 10% of the data using a 50% masking ratio. |
| Hardware Specification | No | The paper does not provide specific details about the hardware used for running experiments, such as specific GPU models, CPU types, or memory configurations. It mentions using 'Res Net-18 backbone' and 'Vi T-Tiny model', but these are model architectures, not hardware specifications. |
| Software Dependencies | No | The paper mentions using 'Res Net-18 backbone', 'Vi T-Tiny model', and 'SGD optimizer'. However, it does not provide specific version numbers for any software libraries, programming languages, or frameworks used (e.g., Python, PyTorch, TensorFlow, CUDA). |
| Experiment Setup | Yes | The network is initially trained for classification over the CIFAR datasets for 182 epochs with an initial learning rate of 0.1 following a cosine decay schedule. For the unlearning procedure, the learning rate is set to 0.02 for our method and 0.013 for the Sal Un method, following the recommendations from the official repository. We set the number of unlearning epochs to 10. Unlearning using the fine-tuning method employs a learning rate of 0.01 for 10 epochs. All methods utilize the SGD optimizer... We set the sparsity at 50%... The learning rate for the Vi T baseline is set to 1 × 10−3... For our method, the learning rate is adjusted to 5 × 10−4. |