AutoLoRa: An Automated Robust Fine-Tuning Framework
Authors: Xilie Xu, Jingfeng Zhang, Mohan Kankanhalli
ICLR 2024 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Extensive empirical evaluations demonstrate that our proposed automated RFT disentangled via the Lo Ra branch (Auto Lo Ra) achieves new state-of-the-art results across a range of downstream tasks. Our comprehensive experimental results validate that our proposed automated robust fine-tuning disentangled via a Lo Ra branch (Auto Lo Ra) is effective in improving adversarial robustness among various downstream tasks. |
| Researcher Affiliation | Academia | 1 School of Computing, National University of Singapore 2 School of Computer Science, The University of Auckland 3 RIKEN Center for Advanced Intelligence Project (AIP) |
| Pseudocode | Yes | Algorithm 1 Automated RFT disentangled via a Lo Ra branch (Auto Lo Ra) |
| Open Source Code | Yes | Our source code is available at the Git Hub. |
| Open Datasets | Yes | We considered six datasets as the downstream tasks. ➊CIFAR-10 with 10 classes and ➋CIFAR-100 (Krizhevsky, 2009) with 100 classes are low-resolution image datasets... ➌Describable textures dataset with 57 classes (DTD-57) (Cimpoi et al., 2014) is a collection of high-resolution textural images... ➍Stanford Dogs dataset with 120 dog categories (DOG-120) (Khosla et al., 2011)... ➎Caltech-UCSD Birds-200-2011 with 200 categories of birds (CUB200) (Wah et al., 2011)... ➏Caltech-256 with 257 classes (Griffin et al., 2007)... |
| Dataset Splits | Yes | We randomly selected 5% of the entire training data as the validation set. |
| Hardware Specification | No | The paper does not provide specific hardware details (e.g., exact GPU/CPU models, processor types with speeds, memory amounts, or detailed computer specifications) used for running its experiments. |
| Software Dependencies | No | The paper does not provide specific ancillary software details (e.g., library or solver names with version numbers like Python 3.8, CPLEX 12.4) needed to replicate the experiment. |
| Experiment Setup | Yes | For the fair comparison, we set maximum training epoch E = 60 following (Liu et al., 2023). We used SGD as the optimizer, froze the weight decay of SGD as 1e 4, and set the rank of the Lo Ra branch rnat = 8 by default. We set α = 1.0 and λmax 2 = 6.0 by default. We randomly selected 5% of the entire training data as the validation set. During training, we used PGD-10 with an adversarial budget of 8/255 and step size of 2/255 to generate the adversarial training and validation data. |