Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in [1].
Understanding Overadaptation in Supervised Fine-Tuning: The Role of Ensemble Methods
Authors: Yifan Hao, Xingyuan Pan, Hanning Zhang, Chenlu Ye, Rui Pan, Tong Zhang
ICML 2025 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We demonstrate that the same holds for language models, and, more strikingly, we observe an overadaptation phenomenon: the ensemble model not only retains general knowledge from the foundation model but also outperforms the fine-tuned model even on the fine-tuning domain itself... supported by empirical experiments consistent with our analysis. Specifically, we start with presenting empirical evidence in Section 3 that highlights the harmful effects of overadaptation and demonstrates the efficiency benefits of ensembling in both improving fine-tuning performance and mitigating forgetting. |
| Researcher Affiliation | Academia | 1University of Illinois Urbana-Champaign, Illinois. |
| Pseudocode | No | The paper describes mathematical frameworks and theoretical analysis in sections 4, 5, and 6, but does not contain structured pseudocode or algorithm blocks. |
| Open Source Code | Yes | We make our implementation publicly available 1. https://github.com/xypan0/LLMForgetting |
| Open Datasets | Yes | Our experiments utilize the Dolly dataset (Conover et al., 2023), a popular instruction-following dataset... The LLMs instruction-following ability is evaluated on MT-Bench (Zheng et al., 2023)... We also assess LLMs general ability on MMLU (Hendrycks et al., 2021) and Commonsense-QA (Talmor et al., 2019). |
| Dataset Splits | Yes | We have a carefully curated instruction-following dataset. The validation dataset consists of multi-turn conversations... our validation dataset contains 600 samples, evenly distributed across the 8 categories in MT-Bench. |
| Hardware Specification | Yes | Our training and evaluation are conducted on 8 NVIDIA H100 GPUs. |
| Software Dependencies | No | We implemented our fine-tuning code based on Huggingface Transformers3 and Accelerate4 libraries, where Fully Sharded Data Parallel (Zhao et al., 2023) is utilized for model parallel training and acceleration. Specific version numbers for these libraries are not provided in the text. |
| Experiment Setup | Yes | We fine-tune the models with a global batch size of 16, and an epoch of 1 using Adam optimizer on 8 GPUs. To select a suitable learning rate and penalty, we search the learning rate on {5 10 6, 2 10 6, 10 6}, and penalty coefficient λ on {10 2, 5 10 3, 2 10 3, 10 3}. We also search the ensemble weight τ uniformly on {0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9}. |