A Boo(n) for Evaluating Architecture Performance
Authors: Ondrej Bajgar, Rudolf Kadlec, Jan Kleindienst
ICML 2018 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We have run several experiments to quantify the scope of the problems outlined in Section 2. We just briefly summarize the main results here for illustration; a more detailed description of the experiments and analysis in the form of an i Python notebook can be found in the Gitlab repository or in Appendix C. |
| Researcher Affiliation | Industry | 1 IBM Watson, Prague AI Research & Development Lab. RK has since moved to Deepmind. |
| Pseudocode | No | The paper does not contain any sections or figures labeled 'Pseudocode' or 'Algorithm', nor does it present structured code-like procedures. |
| Open Source Code | Yes | The data and code for their analysis can be found at http://gitlab.com/obajgar/boon, along with Python functions you can use to calculate Boon. |
| Open Datasets | Yes | We repeatedly trained models from two domains of deep learning: the Res Net (He et al., 2016) on the CIFAR100 dataset (Krizhevsky & Hinton, 2009) to represent Image Recognition and the Attention Sum Reader (AS Reader) (Kadlec et al., 2016) on the Children s Book Test Common Nouns (CBT CN) (Hill et al., 2016) to represent Reading Comprehension. |
| Dataset Splits | No | The paper refers to the use of 'validation set' and 'test set' and discusses their correlation, but it does not specify the explicit training, validation, or test data splits (e.g., percentages or counts) for the datasets used in its experiments (CIFAR100, CBT CN). |
| Hardware Specification | No | The paper mentions training on 'a single GPU' but does not provide specific hardware details such as the GPU model, CPU type, or memory specifications. |
| Software Dependencies | No | The paper mentions 'Python functions', 'i Python notebook', 'pip install boon', and 'numpy', but does not provide specific version numbers for these software components or any other libraries used. |
| Experiment Setup | No | The paper discusses using 'fixed hyperparameters' and 'random hyperparameter sampling' and mentions training time, but it does not provide specific hyperparameter values (e.g., learning rate, batch size, epochs) or other detailed training configurations in the main text. |