Snapshot Ensembles: Train 1, Get M for Free
Authors: Gao Huang, Yixuan Li, Geoff Pleiss, Zhuang Liu, John E. Hopcroft, Kilian Q. Weinberger
ICLR 2017 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We show in a series of experiments that our approach is compatible with diverse network architectures and learning tasks. It consistently yields lower error rates than state-of-the-art single models at no additional training cost, and compares favorably with traditional network ensembles. On CIFAR-10 and CIFAR-100 our Dense Net Snapshot Ensembles obtain error rates of 3.4% and 17.4% respectively. |
| Researcher Affiliation | Academia | Gao Huang , Yixuan Li , Geoff Pleiss Cornell University {gh349, yl2363}@cornell.edu, geoff@cs.cornell.edu Zhuang Liu Tsinghua University liuzhuangthu@gmail.com John E. Hopcroft, Kilian Q. Weinberger Cornell University jeh@cs.cornell.edu, kqw4@cornell.edu |
| Pseudocode | No | The paper describes the method using text and mathematical formulas but does not include a formal pseudocode or algorithm block. |
| Open Source Code | Yes | Code to reproduce results is available at https://github.com/gaohuang/Snapshot Ensemble |
| Open Datasets | Yes | The two CIFAR datasets (Krizhevsky & Hinton, 2009) consist of colored natural images sized at 32 32 pixels. CIFAR-10 (C10) and CIFAR-100 (C100) images are drawn from 10 and 100 classes, respectively. For each dataset, there are 50,000 training images and 10,000 images reserved for testing. We use a standard data augmentation scheme (Lin et al., 2013; Romero et al., 2014; Lee et al., 2015; Springenberg et al., 2014; Srivastava et al., 2015; Huang et al., 2016b; Larsson et al., 2016). SVHN. The Street View House Numbers (SVHN) dataset (Netzer et al., 2011) contains 32 32 colored digit images from Google Street View, with one class for each digit. There are 73,257 images in the training set and 26,032 images in the test set. Tiny Image Net. The Tiny Image Net dataset3 consists of a subset of Image Net images (Deng et al., 2009). Image Net. The ILSVRC 2012 classification dataset (Deng et al., 2009) consists of 1000 images classes, with a total of 1.2 million training images and 50,000 validation images. |
| Dataset Splits | Yes | CIFAR... For each dataset, there are 50,000 training images and 10,000 images reserved for testing. SVHN... There are 73,257 images in the training set and 26,032 images in the test set. Following common practice (Sermanet et al., 2012; Goodfellow et al., 2013; Huang et al., 2016a), we withhold 6,000 training images for validation, and train on the remaining images without data augmentation. Image Net... with a total of 1.2 million training images and 50,000 validation images. |
| Hardware Specification | No | The paper mentions 'even on high performance hardware with GPU acceleration' but does not provide specific details such as GPU models, CPU types, or memory specifications. |
| Software Dependencies | Yes | We run all experiments with Torch 7 (Collobert et al., 2011)2. |
| Experiment Setup | Yes | We test several state-of-the-art architectures, including residual networks (Res Net) (He et al., 2016a), Wide Res Net (Zagoruyko & Komodakis, 2016) and Dense Net (Huang et al., 2016a)... We use a mini batch size of 64.4... We test models with the max learning rate α0 set to 0.1 and 0.2. In both cases, we divide the training process into learning rate cycles. Model snapshots are taken after each learning rate cycle. On CIFAR datasets, the training budget is B = 300 epochs for Dense Net-40 and Dense Net-100, and B = 200 for Res Net and Wide Res Net models. Snapshot variants are trained with M = 6 cycles of B/M = 50 epochs for Dense Nets, and M = 5 cycles of B/M = 40 epochs for Res Nets/Wide Res Nets. SVHN models are trained with a budget of B = 40 epochs (5 cycles of 8 epochs). For Tiny Image Net, we use a training budget of B = 150 (6 cycles of 25 epochs). Finally, Image Net is trained with a budget of B = 90 epochs, and we trained 2 Snapshot variants: one with M = 2 cycles and one with M = 3. |