MetaCoCo: A New Few-Shot Classification Benchmark with Spurious Correlation
Authors: Min Zhang, Haoxuan Li, Fei Wu, Kun Kuang
ICLR 2024 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Extensive experiments on the proposed benchmark are performed to evaluate the state-of-the-art methods in FSC, cross-domain shifts, and self-supervised learning. The experimental results show that the performance of the existing methods degrades significantly in the presence of spurious-correlation shifts. |
| Researcher Affiliation | Academia | Min Zhang1 Haoxuan Li2 Fei Wu1 Kun Kuang1 1Zhejiang University 2Peking University |
| Pseudocode | No | The paper describes methods in text but does not contain structured pseudocode or algorithm blocks. |
| Open Source Code | Yes | We open-source all codes of our benchmark and hope that the proposed Meta Co Co can facilitate future research on spurious-correlation shifts problems in FSC. The code is available at: https://github.com/remi MZ/Meta Co Co-ICLR24. |
| Open Datasets | Yes | In this paper, we present Meta Concept Context (Meta Co Co), a large-scale benchmark with a total of 175,637 images, 155 contexts and 100 classes, with spurious-correlation shifts arising from various contexts in the real-world scenarios. We open-source all codes of our benchmark and hope that the proposed Meta Co Co can facilitate future research on spurious-correlation shifts problems in FSC. The code is available at: https://github.com/remi MZ/Meta Co Co-ICLR24. |
| Dataset Splits | Yes | The first 64 categories with the largest number of samples are used as training data, then 20 categories are selected as testing data, and the last 16 categories are used as validation data. |
| Hardware Specification | No | The paper describes model architectures (Conv64F, ResNet12, ResNet18, ResNet10, WRN-28-10) but does not provide specific hardware details such as GPU/CPU models, memory, or cloud instances used for experiments. |
| Software Dependencies | No | The paper does not specify any software dependencies with version numbers (e.g., programming languages, libraries, or frameworks with their specific versions). |
| Experiment Setup | No | The paper mentions image resizing, number of sampled tasks, and evaluation metrics. However, it does not provide specific hyperparameter values such as learning rates, batch sizes, optimizers, or number of epochs, which are crucial for detailed experimental setup. |