Spectral Adapter: Fine-Tuning in Spectral Space
Authors: Fangzhao Zhang, Mert Pilanci
NeurIPS 2024 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We show through extensive experiments that the proposed fine-tuning model enables better parameter efficiency and tuning performance as well as benefits multi-adapter fusion. |
| Researcher Affiliation | Academia | Fangzhao Zhang Electrical Engineering Stanford University zfzhao@stanford.edu Mert Pilanci Electrical Engineering Stanford University pilanci@stanford.edu |
| Pseudocode | No | The paper does not contain structured pseudocode or algorithm blocks. |
| Open Source Code | Yes | Code is released at https://github.com/pilancilab/spectral_adapter. |
| Open Datasets | Yes | Llama3 8B model with Orca Math dataset [38] and evaluation score on GSM8K benchmark [8]. |
| Dataset Splits | No | The paper mentions training data (Orca Math) and test data (GSM8K) but does not provide explicit training/validation/test splits with percentages or sample counts for all experiments, nor does it specify the method for creating such splits beyond "10K Orca Math data (shuffled)". |
| Hardware Specification | Yes | All experiments are done with NVIDIA RTX A6000 GPU. |
| Software Dependencies | No | The paper mentions software like Pytorch [44] and Hugging Face PEFT library but does not provide specific version numbers for these or other software dependencies. |
| Experiment Setup | Yes | Table 4 shows the hyperparameter setting for our Spectral Adapter A used for fine-tuning De BERTa V3base model in Section 4.1. We set number of diagonal blocks to be 4 and enable block sharing for OFT to maintain similar amount of trainable parameters. |