On sensitivity of meta-learning to support data
Authors: Mayank Agarwal, Mikhail Yurochkin, Yuekai Sun
NeurIPS 2021 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We summarize the worst, average and best accuracies of six meta-learning algorithms on three benchmark datasets (see Section 2.1 for data descriptions) in 1-shot, 5-shot, and 10-shot setting in Tables 1, 2, and 3. All meta-learners are trained using code from the authors or more modern meta-learning libraries [3] (see Appendix A for implementation and additional experimental details). |
| Researcher Affiliation | Collaboration | Mayank Agarwal 1 mayank.agarwal@ibm.com Mikhail Yurochkin 1,2 mikhail.yurochkin@ibm.com Yuekai Sun 3 yuekai@umich.edu IBM Research,1 MIT-IBM Watson AI Lab,2 University of Michigan3. |
| Pseudocode | Yes | Algorithm 1 Finding the worst case support examples |
| Open Source Code | Yes | Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)? [Yes] Material for reproducbility of experiments is currently included in the supplement, and will later be released online |
| Open Datasets | Yes | CIFAR-FS [4] is a dataset of 60000 32 32 RGB images from CIFAR-100 partitioned into 64, 16 and 20 classes for training, validation and testing, respectively. FC-100 [31] is also a derivative of CIFAR-100 with a different partition aimed to reduce semantic overlap between 60 classes assigned for training, 20 for validation, and 20 for testing. Mini Image Net [41] is a subsampled, downsized version of Image Net. It consists of 60000 84 84 RGB images from 100 classes split into 64 for training, 16 for validation, and 20 for testing. |
| Dataset Splits | Yes | CIFAR-FS [4] is a dataset of 60000 32 32 RGB images from CIFAR-100 partitioned into 64, 16 and 20 classes for training, validation and testing, respectively. ... FC-100 ... partitioned into 60 classes assigned for training, 20 for validation, and 20 for testing. ... Mini Image Net ... split into 64 for training, 16 for validation, and 20 for testing. |
| Hardware Specification | Yes | Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] We use machines on internal cluster with a single V-100 GPU to run all our experiments |
| Software Dependencies | No | All meta-learners are trained using code from the authors or more modern meta-learning libraries [3]. ... [3] Sébastien MR Arnold, Praateek Mahajan, Debajyoti Datta, Ian Bunner, and Konstantinos Saitas Zarkias. learn2learn: A library for meta-learning research. ar Xiv preprint ar Xiv:2008.12284, 2020. The paper mentions a specific library but does not provide version numbers for other key software components or the library itself. |
| Experiment Setup | Yes | For evaluation we randomly partition each class in each task into 400 potential support examples composing X and 200 query examples composing D (all datasets have 600 examples per class). ... In our experiments we always run Algorithm 1 for 3 iterations. ... Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See sections 3.1 and 3.3 |