A Zest of LIME: Towards Architecture-Independent Model Distances
Authors: Hengrui Jia, Hongyu Chen, Jonas Guan, Ali Shahin Shamsabadi, Nicolas Papernot
ICLR 2022 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We empirically show that our method, which we call Zest, helps in several tasks that require measurements of model similarity: verifying machine unlearning, and detecting many forms of model reuse, such as model stealing, knowledge distillation, and transfer learning. |
| Researcher Affiliation | Collaboration | Hengrui Jia, Hongyu Chen, Jonas Guan University of Toronto and Vector Institute {nickhengrui.jia, hy.chen}@mail.utoronto.ca, jonas@cs.toronto.edu Ali Shahin Shamsabadi Vector Institute and The Alan Turing Institute a.shahinshamsabadi@turing.ac.uk Nicolas Papernot University of Toronto and Vector Institute nicolas.papernot@utoronto.ca |
| Pseudocode | Yes | Algorithm 1: Proposed model distance approach, Zest |
| Open Source Code | Yes | Code is available at: https://github.com/cleverhans-lab/Zest-Model-Distance |
| Open Datasets | Yes | We validate our proposed method, Zest, in the vision, text, and audio domains using four publicly available classifiers and four datasets: Res Net20 trained on CIFAR-10, Res Net50 trained on CIFAR-100, LSTM trained on AG News, and M5 network trained on Speech Commands. |
| Dataset Splits | No | The paper uses standard datasets like CIFAR-10 and CIFAR-100, which have predefined splits, but does not explicitly state the train/validation/test splits (e.g., percentages or exact counts) within the paper's text for the models themselves. |
| Hardware Specification | No | The paper does not explicitly describe the specific hardware (e.g., GPU/CPU models, memory) used for running its experiments. |
| Software Dependencies | No | The paper mentions using open-source implementations for models and algorithms (e.g., Res Net, LIME), implying software like PyTorch, but does not provide specific version numbers for these software components. |
| Experiment Setup | Yes | In phase 1 of Zest, we select 128 samples randomly from the training set. In phase 1 and phase 2, we follow the same implementation and setup as the original LIME paper. In phase 3, we train linear regression models using the least-squares estimation technique. We use relative Cosine similarity (see Appendix C for details) for computing the distance between global signatures in phase 4. Following the original LIME paper, we set L = 1000. |