Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in Coakley et alK. L. Coakley, T. Snelleman, H. Hoos, and O. E. Gundersen, "The embrace of open science: An analysis of a decade of AI research and 56 800 conference papers," Under Review, 2026..
Machine Unlearning via Simulated Oracle Matching
Authors: Kristian G Georgiev, Roy Rinberg, Sam Park, Shivam Garg, Andrew Ilyas, Aleksander Madry, Seth Neel
ICLR 2025 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Empirically, we use a combination of existing evaluations and a new metric based on the KL-divergence to show that even in nonconvex settings, DMM achieves strong unlearning performance relative to existing algorithms. An added benefit of DMM is that it is a meta-algorithm, in the sense that future advances in data attribution translate directly into better unlearning algorithms, pointing to a clear direction for future progress in unlearning. |
| Researcher Affiliation | Collaboration | 1MIT EECS 2Harvard Business School 3Harvard SEAS 4Stanford CS 5Microsoft Research 6Stanford Statistics |
| Pseudocode | Yes | C PSEUDOCODE C.1 ORACLE MATCHING Algorithm C.1 Oracle Matching (OM) C.2 DATAMODEL DIRECT Algorithm C.2 DM-DIRECT C.3 DATAMODEL MATCHING Algorithm C.3 Datamodel Matching (DMM) |
| Open Source Code | Yes | Implementations for all methods are available at: bit.ly/unlearning-via-simulated-oracles |
| Open Datasets | Yes | We apply different approximate unlearning methods to trained DNNs to unlearn forget sets from CIFAR-10 and Image Net-Living-17. |
| Dataset Splits | Yes | On CIFAR-10, we use 9 different forget sets: sets 1,2,3 are random forget sets of sizes 10,100,1000 respectively; sets 4-9 correspond to semantically coherent subpopulations of examples (e.g., all dogs facing a similar direction) identified using clustering methods. On Image Net Living-17, we use three different forget sets: set 1 is random of size 500; sets 2 and 3 correspond to 200 examples from a certain subpopulation (corresponding to a single original Image Net class) within the Living-17 superclass. We re-train models on random 50% subsets of the full train dataset, and use between 1,000 and 20,000 models. |
| Hardware Specification | No | No specific hardware details (e.g., GPU/CPU models, memory) are mentioned in the paper for the experimental setup. |
| Software Dependencies | No | The paper mentions a GitHub link to a Python script (https://github.com/wbaek/torchskeleton/blob/master/bin/dawnbench/cifar10.py) which likely uses a deep learning framework like PyTorch, but it does not specify any version numbers for Python or any libraries. |
| Experiment Setup | Yes | For CIFAR-10, we train Res Net-9 models5 for 24 epochs with SGD with a batch size of 512, momentum 0.9, and weight decay 5e 4. We set learning rate initially at 0.4, and a single-peak cosine schedule peaking at the 5th epoch. We use a momentum of 0.9 and a weight decay of 5e 4. For Image Net Living17 (Santurkar et al., 2020), we train Res Net-18 models for 25 epochs using SGD with a batch size of 1024, momentum 0.9, and weight decay 5e 4. Label smoothing is set to 0.1. |