Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in Coakley et alK. L. Coakley, T. Snelleman, H. Hoos, and O. E. Gundersen, "The embrace of open science: An analysis of a decade of AI research and 56 800 conference papers," Under Review, 2026..
A Closer Look at Machine Unlearning for Large Language Models
Authors: Xiaojian Yuan, Tianyu Pang, Chao Du, Kejiang Chen, Weiming Zhang, Min Lin
ICLR 2025 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Experimental results across three scenarios, i.e., fictitious unlearning, continual unlearning, and real-world unlearning, demonstrate the effectiveness of our approaches. The code is available at https://github.com/sail-sg/closer-look-LLM-unlearning. |
| Researcher Affiliation | Collaboration | 1University of Science and Technology of China 2Sea AI Lab, Singapore EMAIL; EMAIL |
| Pseudocode | No | The paper describes methods using mathematical formulations and textual explanations but does not include any explicitly labeled pseudocode or algorithm blocks. |
| Open Source Code | Yes | The code is available at https://github.com/sail-sg/closer-look-LLM-unlearning. |
| Open Datasets | Yes | Experimentally, we consider the widely adopted TOFU benchmark (Maini et al., 2024; Zhang et al., 2024a; Jia et al., 2024; Ji et al., 2024; Huang et al., 2024; Liu et al., 2024a) for fictitious unlearning, and extend it to the continual unlearning scenario. Besides, we also conduct evaluations in a more realistic real-world unlearning scenario. |
| Dataset Splits | Yes | It has three levels of tasks, namely forget01, forget05 and forget10, to forget 1%, 5%, and 10% of the constructed data. The complement of each forget set serves as the retain set (i.e., neighbor set). |
| Hardware Specification | Yes | All experiments are conducted on two NVIDIA A100 GPUs with 40GB of memory. |
| Software Dependencies | No | The paper mentions using Llama2-chat-7B and Llama-3-8B-Instruct models, Deep Speed with Ze RO3, and Sentence-BERT and NLI models, but does not provide specific version numbers for any of these software components or libraries. |
| Experiment Setup | Yes | We employ the Adam W optimizer with a weight decay of 0.01, a learning rate of 1 10 5, and an effective batch size of 32 for all experiments, consistent with the settings in (Maini et al., 2024; Zhang et al., 2024a). During unlearning, we fine-tune for 5 epochs, using a linear warm-up learning rate in the first epoch and a linearly decaying learning rate in the subsequent epochs. Following the setup in (Maini et al., 2024; Zhang et al., 2024a), we randomly sample up to 300 question-answer pairs from the dataset for evaluation to improve efficiency. The β in NPO and AP is set to 0.1. The parameter α in MG+GD is set to 0.1 in Section 5.1 and is set to 1.0 in Section 5.2. |