Theoretical and Practical Perspectives on what Influence Functions Do
Authors: Andrea Schioppa, Katja Filippova, Ivan Titov, Polina Zablotskaia
NeurIPS 2023 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | However, recent empirical studies have shown that the existing methods of estimating IF predict the leave-one-out-and-retrain effect poorly. We illustrate this theoretical result with BERT and Res Net models. Another conclusion from the theoretical analysis is that IF are still useful for model debugging and correcting even though some of the assumptions made in prior work do not hold: using natural language processing and computer vision tasks, we verify that mis-predictions can be successfully corrected by taking only a few fine-tuning steps on influential examples. |
| Researcher Affiliation | Collaboration | Andrea Schioppa1 Katja Filippova1 Ivan Titov2,3 Polina Zablotskaia1 1Google Deep Mind 2University of Edinburgh 3University of Amsterdam |
| Pseudocode | No | The paper does not contain any structured pseudocode blocks or algorithms. |
| Open Source Code | No | The paper does not contain an explicit statement about releasing the source code for the described methodology, nor does it provide a link to a code repository. |
| Open Datasets | Yes | We consider binary classification for nlp, where we fine-tune BERT on SST2; for computer vision we consider multi-class classification where we train from scratch Res Net on CIFAR10. We use the notoriously noisy Wikipedia Toxicity Subtypes dataset [WTD17]... |
| Dataset Splits | Yes | We consider binary classification for nlp, where we fine-tune BERT on SST2; for computer vision we consider multi-class classification where we train from scratch Res Net on CIFAR10. The best checkpoint was selected on validation-set accuracy evaluated every 500 steps; |
| Hardware Specification | Yes | Res Net was trained on a single V100 with the hyper-parameters in Table 3; training data was augmented using torchvision using the transformations transforms.Random Crop(32, padding=4), and transforms.Random Horizontal Flip(). The Vi T was trained on a single V100 with the hyper-parameters in Table 6; we used the same data augmentation as in the case of Res Net. BERT was trained on 8 TPUv3 cores using the hyper-parameters in Table 4. T5 was trained on a GPU V100 using the hyper-parameters in Table 5. |
| Software Dependencies | No | The paper mentions software like 'torchvision' and refers to 'ABIF [SZTS22]', but it does not provide specific version numbers for these or other software dependencies. |
| Experiment Setup | Yes | Res Net was trained on a single V100 with the hyper-parameters in Table 3; training data was augmented using torchvision using the transformations transforms.Random Crop(32, padding=4), and transforms.Random Horizontal Flip(). For BERT we used the hyper-parameters in Table 4. For T5 we used the hyper-parameters in Table 5. For ViT we used the hyper-parameters in Table 6. |