CS-Shapley: Class-wise Shapley Values for Data Valuation in Classification
Authors: Stephanie Schoch, Haifeng Xu, Yangfeng Ji
NeurIPS 2022 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Further, our experiments on two benchmark evaluation tasks (data removal and noisy label detection) and four classifiers demonstrate the effectiveness of CS-SHAPLEY over existing methods. |
| Researcher Affiliation | Academia | Department of Computer Science, University of Virginia, Charlottesville, VA 22904 Department of Computer Science, University of Chicago, Chicago, IL 60637 |
| Pseudocode | Yes | The detailed implementation of our algorithm can be found in the pseudo-code deferred to Appendix A. |
| Open Source Code | Yes | 1Code is available at https://github.com/stephanieschoch/cs-shapley |
| Open Datasets | Yes | We use nine benchmark datasets: Diabetes, CPU, Click, Covertype, CIFAR10 (binarized), FMNIST (binarized), MNIST (multi-class and binarized versions, denoted using -2 and -10, respectively), and Phoneme. |
| Dataset Splits | Yes | The value function of prior Shapley-based data valuation methods has typically been defined as the predictive accuracy over the entire development set. However, in the context of valuing data for learning models on classification tasks, this may have limited ability to differentiate helpful or harmful training instances. ... Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See section 5 and Appendix A. |
| Hardware Specification | Yes | Computational cost: All experiments were run on an internal cluster running Ubuntu 20.04. GPUs used were NVIDIA GeForce RTX 3090 with 24GB GDDR6X memory and/or NVIDIA GeForce GTX 1080 Ti with 11GB GDDR5X memory. CPUs used were Intel Xeon Gold 6248 CPUs. The average runtimes for the most computationally intensive tasks were: TMC-Shapley (24 hours), Beta Shapley (48 hours), CS-SHAPLEY (24 hours). |
| Software Dependencies | No | All classifiers were implemented using scikit-learn [1] with the exception of the multi-layer perceptron (MLP) which was implemented in PyTorch [19]. Specific version numbers for scikit-learn or PyTorch are not provided in the given text. |
| Experiment Setup | Yes | For Beta Shapley, we used the best and β values suggested in the original paper, which were also verified by our preliminary hyperparameter search. ... Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See section 5 and Appendix A. |