Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in [1].
CONTRAST: Continual Multi-source Adaptation to Dynamic Distributions
Authors: Sk Miraj Ahmed, Fahim Faisal Niloy, Xiangyu Chang, Dripta S. Raychaudhuri, Samet Oymak, Amit Roy-Chowdhury
NeurIPS 2024 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Experimental analysis on diverse datasets demonstrates that the combination of multiple source models does at least as well as the best source (with hindsight knowledge), and performance does not degrade as the test data distribution changes over time (robust to forgetting). |
| Researcher Affiliation | Collaboration | 1University of California, Riverside, 2Brookhaven National Laboratory, 3AWS AI Labs, 4University of Michigan, Ann Arbor |
| Pseudocode | Yes | Algorithm 1: Overview of CONTRAST |
| Open Source Code | No | Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [No] |
| Open Datasets | Yes | Datasets. We demonstrate the efficacy of our approach using both static target distribution and dynamic target data distributions. For static case, we employ the Digits and Office-Home datasets [42]. For the dynamic case, we utilize CIFAR-100C and CIFAR-10C [43]. |
| Dataset Splits | No | No explicit training/validation/test splits are provided for the datasets used in the experiments with specific percentages or counts. The theoretical section mentions 'validation(ν) distribution D' but this does not specify the splits used in the empirical evaluations. |
| Hardware Specification | No | No specific hardware details (e.g., GPU/CPU models, memory amounts) are provided for running the experiments. The NeurIPS checklist mentions 'GPU' generally but no specific models or configurations. |
| Software Dependencies | No | The paper mentions software components like 'Adam optimizer' and uses frameworks implicitly (e.g., ResNet-18 models), but it does not provide specific version numbers for programming languages, libraries, or frameworks (e.g., Python, PyTorch, CUDA versions). |
| Experiment Setup | Yes | We use Res Net-18 [44] model for all our experiments. For solving the optimization of Eq. (4), we first initialize the combination weights using Eq. (6) and calculate the optimal learning rate using Eq. (7). After that, we use 5 iterations to update the combination weights using SGD optimizer and the optimal learning rate. For all the experiments we use a batch size of 128, as used by Tent [3]. For more details on implementation and experimental setting see Appendix. |