Realistic evaluation of transductive few-shot learning
Authors: Olivier Veilleux, Malik Boudiaf, Pablo Piantanida, Ismail Ben Ayed
NeurIPS 2021 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We evaluate experimentally state-of-the-art transductive few-shot methods over 3 widely used datasets, and observe that the performances decrease by important margins, albeit at various degrees, when dealing with arbitrary class distributions. |
| Researcher Affiliation | Academia | Olivier Veilleux ÉTS Montreal Malik Boudiaf * ÉTS Montreal Pablo Piantanida L2S, Centrale Supélec CNRS Université Paris-Saclay Ismail Ben Ayed ÉTS Montreal |
| Pseudocode | No | The paper describes methods and procedures in text but does not include any clearly labeled pseudocode or algorithm blocks. |
| Open Source Code | Yes | Our code is publicly available at https://github.com/oveilleux/Realistic_Transductive_Few_Shot. |
| Open Datasets | Yes | Datasets We use three standard benchmarks for few-shot classification: mini-Imagenet [46], tiered Imagenet [30] and Caltech-UCSD Birds 200 (CUB) [47]. |
| Dataset Splits | Yes | We used different Dirichlet s concentration parameter a for validation and testing. The validation-task generation is based on a random sampling within the simplex (i.e Dirichlet with a = 1K). Testing-task generation uses a = 2 1K to reflect the fact that extremely imbalanced tasks (i.e., only one class is present in the task) are unlikely to happen in practical scenarios; see Figure 1 for visualization. ... For all methods, hyper-parameter tuning is performed on the validation set of each dataset, using the validation sampling described in the previous paragraph. |
| Hardware Specification | Yes | All the experiments have been executed on a single GTX 1080 Ti GPU. |
| Software Dependencies | No | The paper mentions software components like PyTorch and libraries used implicitly by the described models (e.g., Res Net-18, WRN28-10), but it does not specify explicit version numbers for these software dependencies (e.g., 'PyTorch 1.9'). |
| Experiment Setup | Yes | The feature extractors are trained for 90 epochs, using a learning rate initialized to 0.1 and divided by 10 at epochs 45 and 66. We use a batch size of 256 for Res Net-18 and of 128 for WRN28-10. During training, color jitter, random croping and random horizontal flipping augmentations are applied. |