PACE: Marrying generalization in PArameter-efficient fine-tuning with Consistency rEgularization
Authors: Yao Ni, Shan Zhang, Piotr Koniusz
NeurIPS 2024 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Experimental evidence supports our theories. PACE surpasses existing PEFT methods in visual adaptation tasks (VTAB-1k, FGVC, few-shot learning, domain adaptation) showcasing its potential for resource-efficient finetuning. It also improves Lo RA in text classification (GLUE) and mathematical reasoning (GSM-8K). |
| Researcher Affiliation | Collaboration | Yao Ni Shan Zhang , Piotr Koniusz , , The Australian National University Data61 CSIRO Australian Institute for Machine Learning, The University of Adelaide yao.ni@anu.edu.au shan.zhang@adelaide.edu.au piotr.koniusz@data61.csiro.au |
| Pseudocode | No | The paper describes the pipeline and steps involved in PACE, particularly in Section 3.5 and Figure 2, but it does not present a clearly labeled pseudocode block or algorithm box. |
| Open Source Code | Yes | The code is available at github.com/Maxwell Yao Ni/PACE. |
| Open Datasets | Yes | Datasets and evluations. VTAB-1K comprises 19 datasets organized into (i) Natural images, (ii) Specialized datasets (remote sensing, medical) and (iii) Structured datasets (scene structure) domains. Each dataset has 1K training examples... Few-shot learning involves 5 fine-grained datasets: FGVC-Aircraft [46], Food101 [4], Oxford Flowers102 [51], Oxford Pets [53] and Stanford Cars [35]... FGVC includes 5 fine-grained datasets: CUB200-2011 [69], NABirds [66], Oxford Flowers [51], Stanford Dogs [10] and Stanford Cars [35]... For domain adaptation, following [82, 7], we train on Image Net [11]... We evaluate on GLUE [70] for text classification and GSM-8K [9] for mathematical reasoning. |
| Dataset Splits | Yes | VTAB-1K... Following [78, 28], we use the provided 800-200 train split for hyperparameter selection, evaluate using the full training set and report average accuracy across three trails... Few-shot learning... tune hyperparameters using validation and report average test accuracy over three random seeds... For domain adaptation... use the validation split by [82] for hyperparameter selection and report the results on the official validation set and 4 out-of-domain datasets... |
| Hardware Specification | Yes | All experiments used an NVIDIA H100 GPU. |
| Software Dependencies | No | The paper mentions using "Adam optimizer [32]" but does not specify its version number or versions for other key software components like programming languages (e.g., Python) or deep learning frameworks (e.g., PyTorch, TensorFlow, CUDA). |
| Experiment Setup | Yes | Models are fine-tuned for 300 epochs on VTAB-1K and 100 epochs on other vision adaptation tasks, with batch size 64. For text classification we follow [26]... We use the Adam optimizer [32] with cosine learning rate decay and linear warm-up (first 10 epochs)... Building on the strong Lo RAmul+VPTadd, we use the grid search for our λ and σ, following strategies from previous studies [28, 41, 26]. ... Table 16, 17, 18 and 19 present the hyperparameters and number of trainable parameters used in our strong baseline for VTAB-1K, few-shot learning, FGVC and domain adaptation tasks. |