Efficient Test-Time Model Adaptation without Forgetting
Authors: Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Yaofo Chen, Shijian Zheng, Peilin Zhao, Mingkui Tan
ICML 2022 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Extensive experiments on CIFAR-10-C, Image Net-C, and Image Net-R verify the effectiveness of our proposed method. |
| Researcher Affiliation | Collaboration | 1School of Software Engineering, South China University of Technology, China 2Pazhou Laboratory, China 3Tencent AI Lab, China 4National University of Singapore, Singapore 5Key Laboratory of Big Data and Intelligent Robot, Ministry of Education, China. |
| Pseudocode | Yes | The pseudo-code of EATA is summarized in Algorithm 1. |
| Open Source Code | Yes | Code is available at https://github.com/mr-eggplant/EATA. |
| Open Datasets | Yes | We conduct experiments on three benchmarks datasets for OOD generalization, i.e., CIFAR-10-C, Image Net-C (Hendrycks & Dietterich, 2019) (...) and Image Net-R (Hendrycks et al., 2021). (...) The models are trained on CIFAR-10 or Image Net training set and then tested on clean or the above OOD test sets. |
| Dataset Splits | Yes | We conduct experiments on three benchmarks datasets for OOD generalization, i.e., CIFAR-10-C, Image Net-C (...) and Image Net-R (...). (...) We report the comparisons on Image Net-C with the highest severity level 5 in Table 2 and put more results of other severity levels 1-4 into Supplementary due to the page limitation. |
| Hardware Specification | Yes | For Res Net-50 on Image Net-C (Gaussian noise, level=5, 50,000 images) with one Tesla V100 GPU, the actual run time is 113s for Tent (28.6% accuracy) and 102s for EATA (35.1% accuracy). |
| Software Dependencies | No | The paper mentions "Py Torch version" but does not specify a version number for PyTorch or any other software dependencies. |
| Experiment Setup | Yes | For test-time adaptation, we use SGD as the update rule, with a momentum of 0.9, batch size of 64, and learning rate of 0.005/0.00025 for CIFAR10/Image Net (following Tent and MEMO). The entropy constant E0 in Eqn. (3) is set to 0.4 ln C, where C is number of task classes. The ϵ in Eqn. (6) is set to 0.4/0.05 for CIFAR10/Image Net. The trade-off parameter β in Eqn. (8) is set to 1/2,000 for CIFAR-10/Image Net to make two losses have the similar magnitude. We use 2,000 samples to calculate ω(θi) in Eqn. (9) which takes 2,000 additional forward-andbackward passes. The moving average factor α in Eqn. (4) is set to 0.1. |