DisCEdit: Model Editing by Identifying Discriminative Components
Authors: Chaitanya Murti, Chiranjib Bhattacharyya
NeurIPS 2024 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | 6 Empirical Evaluations In this section, we empirically study the effectiveness of identifying discriminative filters for model editing tasks, specifically structured pruning and class unlearning. Additional experimental details are given in Appendix E. Experiments showing that class-conditional feature map distributions are non-gaussian, the effectiveness of variants of witness functions, the effectiveness of sparsity in class forgetting, and other ablations, are provided in Appendix E. Our experiment setup is provided in Appendix F, along with baseline accuracies of all models, shown in Table 11. |
| Researcher Affiliation | Academia | Chaitanya Murti Robert Bosch Centre for Cyberphysical Systems Indian Institute of Science mchaitanya@iisc.ac.in Chiranjib Bhattacharyya Computer Science and Automation Robert Bosch Centre for Cyberphysical Systems Indian Institute of Science chiru@iisc.ac.in |
| Pseudocode | Yes | Algorithm 1: DISCEDIT-X Input: Class conditional distributions Dc and class-complements D c for all c [C], Pretrained CNN with parameters W = (W1, , WL), layerwise sparsity budgets Bl, witness function f for l [L] do Set Sl = [sl 1, , sl Nl] = 0Nl if X is SP then For each j, compute rl j using (9). if j sort Bl({rl j} Nl j=1) then Set sl j = 1 if X is U and Forget Class is c then For each j, compute rl j using (10). if j sort Bl({rl j} Nl j=1) then Set sl j = 0 Output: Binary masks S1, , SL |
| Open Source Code | Yes | 1Our code is available at: https://github.com/chaimurti/Dis CEdit |
| Open Datasets | Yes | We apply DISCEDIT-U to selective class forgetting on models trained on CIFAR10 and CIFAR100...Similarly, on Structured pruning problems...on Res Net50 on Imagenet... All CIFAR10 and CIFAR100 models were obtained from: https://github.com/chenyaofo/pytorch-cifar-models. Res Net50 trained on Imagenet was obtained from: https://drive.google.com/drive/folders/1bdZlvKUUu0rXqMYAtIr0ynHQHuEWDI, which in turn comes from: https://github.com/Eclipsess/CHIP_NeurIPS2021?tab=readme-ov-file |
| Dataset Splits | Yes | Table 12: Breakdown of dataset splits used in our experiments. Dataset Training Set TV Distance Set Test Set CIFAR10 Not used 4000 images from test set 6000 images from Test set CIFAR100 Not used 4000 images from test set 6000 images from Test set Imagenet Not used 30000 images from Validation set 20000 images from Val. set |
| Hardware Specification | Yes | 1. Server computer with 2 NVIDIA RTX3090Ti GPUs with Intel i9-12700 processors, running Ubuntu 20.04, with Python 3.11 and CUDA Tools 10.2 with Py Torch 2.0.1. 2. Desktop computer with 1 NVIDIA RTX3070Ti GPUs with Intel i7-10700 processor, running Ubuntu 22.04, with Python 3.11 and CUDA Tools 11.7 with Py Torch 2.0.1. |
| Software Dependencies | Yes | 1. Server computer with 2 NVIDIA RTX3090Ti GPUs with Intel i9-12700 processors, running Ubuntu 20.04, with Python 3.11 and CUDA Tools 10.2 with Py Torch 2.0.1. 2. Desktop computer with 1 NVIDIA RTX3070Ti GPUs with Intel i7-10700 processor, running Ubuntu 22.04, with Python 3.11 and CUDA Tools 11.7 with Py Torch 2.0.1. |
| Experiment Setup | Yes | F.1.4 Hyperparameter Details In this section, we detail the hyperparameters used when fine-tuning pruned models. We present the hyperparameters for CIFAR10 and Imagenet models only, as we did not fine-tune models that used CIFAR100. CIFAR10 Fine-tuning We detail the hyperparameters used in our CIFAR10 experiments below. 1. Batch Size: 128 2. Epochs: 50 3. Learning Rate: .001 4. Weight Decay: .0005 5. Momentum paramters: .9 6. Optimizer: SGD |