Lipsum-FT: Robust Fine-Tuning of Zero-Shot Models Using Random Text Guidance
Authors: Giung Nam, Byeongho Heo, Juho Lee
ICLR 2024 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Extensive experiments conducted on distribution shift scenarios in Domain Net and Image Net confirm the superiority of our proposed Lipsum-FT approach over existing robust fine-tuning methods. |
| Researcher Affiliation | Collaboration | Giung Nam1 Byeongho Heo2 Juho Lee1,3 1KAIST AI 2NAVER AI Lab 3AITRICS |
| Pseudocode | No | No pseudocode or algorithm blocks labeled 'Pseudocode' or 'Algorithm' were found. |
| Open Source Code | Yes | Code is available at https://github.com/cs-giung/lipsum-ft. |
| Open Datasets | Yes | Image Net (IN; Russakovsky et al., 2015)... Domain Net (Peng et al., 2019) |
| Dataset Splits | Yes | We also evaluate fine-tuned models every 1000 iterations and choose the model that demonstrates the highest performance on the reference validation data, which corresponds to Domain Net-R for the Domain Net scenario and Image Net for the Image Net scenario. |
| Hardware Specification | Yes | All experiments are carried out using eight TPUv2 or TPUv3 cores, supported by TPU Research Cloud. |
| Software Dependencies | No | The paper lists software components but does not provide specific version numbers for them. For example, it mentions 'JAX (Bradbury et al., 2018)' but not 'JAX 0.3.25'. |
| Experiment Setup | Yes | Unless specified, the mini-batch size was set to 256, the peak learning rate was configured as 1e-05 with the initial 10% of steps were dedicated to the warm-up phase. |