Sharpness-diversity tradeoff: improving flat ensembles with SharpBalance
Authors: Haiquan Lu, Xiaotian Liu, Yefan Zhou, Qunli Li, Kurt Keutzer, Michael W. Mahoney, Yujun Yan, Huanrui Yang, Yaoqing Yang
NeurIPS 2024 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Empirically, we conducted comprehensive evaluations in various data sets (CIFAR-10, CIFAR-100, Tiny Image Net) and showed that Sharp Balance not only effectively improves the sharpness-diversity trade-off, but also significantly improves ensemble performance in ID and OOD scenarios. |
| Researcher Affiliation | Academia | 1 Nankai University 2 Dartmouth College 3 University of California San Diego 4 University of California at Berkeley 5 International Computer Science Institute 6 Lawrence Berkeley National Laboratory |
| Pseudocode | No | The paper describes the Sharp Balance method in text and with a system diagram (Figure 6) but does not include a formally labeled 'Pseudocode' or 'Algorithm' block. |
| Open Source Code | Yes | Our code has been made open-source. https://github.com/haiquanlu/Sharp Balance |
| Open Datasets | Yes | We evaluated this trade-off using a variety of image classification datasets, including CIFAR-10, CIFAR-100 [Krizhevsky, 2009], Tiny Image Net [Le and Yang, 2015], and their corrupted versions [Hendrycks and Dietterich, 2019b]. |
| Dataset Splits | Yes | In addition, we use 10% of the training set as the validation set for selecting ρ and k based on the ensemble s performance. |
| Hardware Specification | Yes | All codes are implemented in Py Torch, and the experiments are conducted on 3 Nvidia Quadro RTX 6000 GPUs for training an ensemble of 3 models. |
| Software Dependencies | No | All codes are implemented in Py Torch. No specific version number for PyTorch or other libraries is provided. |
| Experiment Setup | Yes | We use a batch size of 128, a momentum of 0.9, and a weight decay of 0.0005 for model training. |