Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in [1].
A Zest of LIME: Towards Architecture-Independent Model Distances
Authors: Hengrui Jia, Hongyu Chen, Jonas Guan, Ali Shahin Shamsabadi, Nicolas Papernot
ICLR 2022 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We empirically show that our method, which we call Zest, helps in several tasks that require measurements of model similarity: verifying machine unlearning, and detecting many forms of model reuse, such as model stealing, knowledge distillation, and transfer learning. |
| Researcher Affiliation | Collaboration | Hengrui Jia, Hongyu Chen, Jonas Guan University of Toronto and Vector Institute EMAIL, EMAIL Ali Shahin Shamsabadi Vector Institute and The Alan Turing Institute EMAIL Nicolas Papernot University of Toronto and Vector Institute EMAIL |
| Pseudocode | Yes | Algorithm 1: Proposed model distance approach, Zest |
| Open Source Code | Yes | Code is available at: https://github.com/cleverhans-lab/Zest-Model-Distance |
| Open Datasets | Yes | We validate our proposed method, Zest, in the vision, text, and audio domains using four publicly available classifiers and four datasets: Res Net20 trained on CIFAR-10, Res Net50 trained on CIFAR-100, LSTM trained on AG News, and M5 network trained on Speech Commands. |
| Dataset Splits | No | The paper uses standard datasets like CIFAR-10 and CIFAR-100, which have predefined splits, but does not explicitly state the train/validation/test splits (e.g., percentages or exact counts) within the paper's text for the models themselves. |
| Hardware Specification | No | The paper does not explicitly describe the specific hardware (e.g., GPU/CPU models, memory) used for running its experiments. |
| Software Dependencies | No | The paper mentions using open-source implementations for models and algorithms (e.g., Res Net, LIME), implying software like PyTorch, but does not provide specific version numbers for these software components. |
| Experiment Setup | Yes | In phase 1 of Zest, we select 128 samples randomly from the training set. In phase 1 and phase 2, we follow the same implementation and setup as the original LIME paper. In phase 3, we train linear regression models using the least-squares estimation technique. We use relative Cosine similarity (see Appendix C for details) for computing the distance between global signatures in phase 4. Following the original LIME paper, we set L = 1000. |