Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in [1].
Active Test-Time Adaptation: Theoretical Analyses and An Algorithm
Authors: Shurui Gui, Xiner Li, Shuiwang Ji
ICLR 2024 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Extensive experimental results confirm consistency with our theoretical analyses and show that the proposed ATTA method yields substantial performance improvements over TTA methods while maintaining efficiency and shares similar effectiveness to the more demanding active domain adaptation (ADA) methods. |
| Researcher Affiliation | Academia | Shurui Gui Texas A&M University College Station, TX 77843 EMAIL Xiner Li* Texas A&M University College Station, TX 77843 EMAIL Shuiwang Ji Texas A&M University College Station, TX 77843 EMAIL |
| Pseudocode | Yes | Algorithm 1 SIMATTA: A SIMPLE ATTA ALGORITHM |
| Open Source Code | Yes | Our code is available at https://github.com/divelab/ATTA. |
| Open Datasets | Yes | We employ PACS (Li et al., 2017), VLCS (Fang et al., 2013), Office-Home (Venkateswara et al., 2017), and Tiny-Image Net-C datasets for our evaluations. |
| Dataset Splits | No | The paper specifies how source and test domains are chosen (e.g., 'For PACS and Office-Home, we use domains "photos" and "real" as the source domains'), and describes data stream orders, but does not provide explicit numerical training, validation, and test dataset splits with percentages or counts for reproducibility of data partitioning in the traditional sense. |
| Hardware Specification | Yes | The Ubuntu server includes 112 Intel(R) Xeon(R) Gold 6258R CPU @2.70GHz, 1.47TB memory, and NVIDIA A100 80GB PCIe graphics cards. |
| Software Dependencies | No | The paper mentions using 'Py Torch (Paszke et al., 2019)' and 'scikit-learn (Pedregosa et al., 2011)' but does not provide specific version numbers for these libraries to ensure reproducible software dependencies. |
| Experiment Setup | Yes | Source domain pre-training. For the PACS and VLCS datasets, models are fine-tuned on the selected source domains for 3,000 iterations. The Adam optimizer is utilized with a learning rate of 10 4. In contrast, for the Office-Home dataset, the model is fine-tuned for a longer duration of 10,000 iterations with a slightly adjusted learning rate of 5 10 5. |