Regularization in ResNet with Stochastic Depth
Authors: Soufiane Hayou, Fadhel Ayed
NeurIPS 2021 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We introduce a new algorithm called Sense Mode to compute the survival rates under a fixed training budget and provide a series of experiments that validates our Budget hypothesis introduced in Section 5. The objective of this section is two-fold: we empirically verify the theoretical analysis developed in sections 3 and 4 with a Vanilla Res Net model on a toy regression task; we also empirically validate the Budget Hypothesis on the benchmark datasets CIFAR-10 and CIFAR-100 [Krizhevsky et al., 2009]. |
| Researcher Affiliation | Collaboration | Soufiane Hayou Department of Statistics University of Oxford United Kingdom Fadhel Ayed Huawei Technologies France |
| Pseudocode | No | The paper describes algorithmic steps for 'Sense Mode' in prose but does not provide a formally structured pseudocode or algorithm block. |
| Open Source Code | Yes | Notebooks and code to reproduce all experiments, plots and tables presented are available in the supplementary material. |
| Open Datasets | Yes | we also empirically validate the Budget Hypothesis on the benchmark datasets CIFAR-10 and CIFAR-100 [Krizhevsky et al., 2009]. |
| Dataset Splits | No | The paper mentions benchmark datasets CIFAR-10 and CIFAR-100 but does not explicitly state the training, validation, and test splits (e.g., percentages, sample counts, or specific predefined splits) used for the experiments. |
| Hardware Specification | No | The paper does not provide specific details on the hardware (e.g., GPU/CPU models, memory, cloud instances) used for running the experiments. |
| Software Dependencies | No | The paper mentions building on an 'open-source implementation of standard Res Net4' (linking to a PyTorch repository) but does not list specific software dependencies with version numbers (e.g., Python, PyTorch, CUDA versions). |
| Experiment Setup | Yes | Implementation details: Vanilla Stable Res Net is composed of identical residual blocks each formed of a Linear layer followed by Re LU. Res Net110 follows [He et al., 2016, Huang et al., 2016]; it comprises three groups of residual blocks; each block consists of a sequence Convolution Batch Norm-Re LU-Convolution-Batch Norm. We use the adjective "Stable" (Stable Vanilla Res Net, Stable Res Net110) to indicate that we scale the blocks using a factor 1/ L as described in Section 3. ... We compare the three modes: Uniform, Linear, and Sense Mode on two benchmark datasets using a grid survival proportions. |