Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in [1].
Certified Defenses for Data Poisoning Attacks
Authors: Jacob Steinhardt, Pang Wei W. Koh, Percy S. Liang
NeurIPS 2017 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Empirically, we find that even under a simple defense, the MNIST-1-7 and Dogfish datasets are resilient to attack, while in contrast the IMDB sentiment dataset can be driven from 12% to 23% test error by adding only 3% poisoned data. |
| Researcher Affiliation | Academia | Jacob Steinhardt Stanford University EMAIL Pang Wei Koh Stanford University EMAIL Percy Liang Stanford University EMAIL |
| Pseudocode | Yes | Algorithm 1 Online learning algorithm for generating an upper bound and candidate attack. |
| Open Source Code | Yes | The code and data for replicating our experiments is available on Git Hub (http: //bit.ly/gt-datapois) and Codalab Worksheets (http://bit.ly/cl-datapois). |
| Open Datasets | Yes | For MNIST-1-7, following Biggio et al. (2012), we considered binary classification between the digits 1 and 7; this left us with n = 13007 training examples of dimension 784. For Dogfish, which is a binary classification task, we used the same Inception-v3 features as in Koh and Liang (2017)... We ran both the upper bound relaxation and the IQP solver on two text datasets, the Enron spam corpus (Metsis et al., 2006) and the IMDB sentiment corpus (Maas et al., 2011). |
| Dataset Splits | No | The paper provides training set sizes and mentions evaluation on a test set, but it does not specify explicit train/validation/test dataset splits (e.g., percentages, counts, or explicit validation set usage) needed to reproduce the data partitioning. |
| Hardware Specification | No | The paper does not provide specific hardware details (e.g., GPU/CPU models, memory amounts) used for running its experiments. |
| Software Dependencies | No | We used a combination of CVXPY (Diamond and Boyd, 2016), YALMIP (Löfberg, 2004), Se Du Mi (Sturm, 1999), and Gurobi (Gurobi Optimization, Inc., 2016) to solve the optimization. While tools are mentioned, specific version numbers are not provided for them. |
| Experiment Setup | Yes | Here ry, sy are thresholds (e.g., chosen so that 30% of the data is removed). |