Improving Simple Models with Confidence Profiles
Authors: Amit Dhurandhar, Karthikeyan Shanmugam, Ronny Luss, Peder A. Olsen
NeurIPS 2018 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | The value of our method is first demonstrated on CIFAR-10, where our weighting method significantly improves (3-4%) networks with only a fraction of the number of Resnet blocks of a complex Resnet model. We further demonstrate operationally significant results on a real manufacturing problem, where we dramatically increase the test accuracy of a CART model (the domain standard) by roughly 13%. |
| Researcher Affiliation | Industry | Amit Dhurandhar* IBM Research, Yorktown Heights, NY adhuran@us.ibm.com Karthikeyan Shanmugam* IBM Research, Yorktown Heights, NY karthikeyan.shanmugam2@ibm.com Ronny Luss IBM Research, Yorktown Heights, NY rluss@us.ibm.com Peder Olsen IBM Research, Yorktown Heights, NY pederao@us.ibm.com |
| Pseudocode | Yes | Algorithm 1 Prof Weight; Algorithm 2 AUC Weight Computation; Algorithm 3 Neural Network Weight Computation |
| Open Source Code | No | The paper states, 'Code is available at https://github.ibm.com/Karthikeyan-Shanmugam2/Transfer/blob/ master/README.md'. This URL points to an internal IBM GitHub instance, which is not publicly accessible. |
| Open Datasets | Yes | We used the python version from https://www.cs.toronto.edu/ kriz/cifar.html. |
| Dataset Splits | Yes | We split the available 50000 training samples from the CIFAR-10 dataset into training set 1 consisting of 30000 examples and training set 2 consisting of 20000 examples. We split the 10000 test set into a validation set of 500 examples and a holdout test set of 9500 examples. |
| Hardware Specification | No | The paper does not provide specific hardware details such as GPU models, CPU types, or memory amounts used for running the experiments. It only generally refers to the use of 'TensorFlow authors' implementation, which implies computational resources but without specification. |
| Software Dependencies | No | The paper mentions using 'the popular implementation of the Residual Network Model available from the Tensor Flow authors' but does not specify a version number for TensorFlow or any other software dependencies. |
| Experiment Setup | Yes | Simple Models Training: Each of the simpler models are trained only on training set 2 consisting of 20000 samples for 500 epochs. All training hyperparameters are set to be the same as in the previous cases. |