Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in Coakley et alK. L. Coakley, T. Snelleman, H. Hoos, and O. E. Gundersen, "The embrace of open science: An analysis of a decade of AI research and 56 800 conference papers," Under Review, 2026..
Theoretical Investigation of Adafactor for Non-Convex Smooth Optimization
Authors: Yusu Hong, Junhong Lin
NeurIPS 2025 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We finally provide some simple numerical experiments on natural language processing to complement our theoretical results. Many existing works, such as [38, 51, 32, 53], have empirically demonstrated the convergence of Adafactor, showing that it achieves comparable performance to Adam in training NLP models. While our main contribution lies in theoretical parts, we also test our hyper-parameter setups in the full fine-tuning (FFT) scenario. We train BERT-Base and BERT-Large on GLUE/MNLI and GPT-2 on Book Corpus dataset. |
| Researcher Affiliation | Academia | Yusu Hong Center for Data Science and School of Mathematical Sciences Zhejiang University EMAIL Junhong Lin Center for Data Science Zhejiang University EMAIL |
| Pseudocode | Yes | Algorithm 1 Adafactor Input: Horizon T, initialization X1 Rn m, R0 = 0m, C0 = 0 n , step-size parameters {ρk}k 1, decay rates {β2,k}k 1 [0, 1), regularization constant ϵ1 > 0, clipping threshold d. for k = 1, , T do Draw a random sample Zk and Gk = g(Xk, Zk); Rk = β2,k Rk 1 + (1 β2,k)(Gk Gk + ϵ11n1 m)1m; Ck = β2,k Ck 1 + (1 β2,k)1 n (Gk Gk + ϵ11n1 m); Wk = (Rk Ck)/(1 n Rk); Uk = Gk/ Wk; ηk = ρk/ max{1, RMS(Uk)/d}; Xk+1 = Xk ηk Gk/ Wk; end for |
| Open Source Code | No | Answer: [No] Justification: Our code is based on Pytorch package which is standard. In addition, we have clarified the detailed experimental setup in our paper and the experiments are easy to reproduce. |
| Open Datasets | Yes | We train BERT-Base and BERT-Large on GLUE/MNLI and GPT-2 on Book Corpus dataset. |
| Dataset Splits | No | The paper does not explicitly mention the dataset splits for GLUE/MNLI or Book Corpus dataset, only that they were used for training. While these are standard benchmarks, the specific split percentages or methodology are not detailed. |
| Hardware Specification | Yes | All the experiments are conducted using the fairseq implementation of Adafactor 4 and the Hugging Face implementation of Adam on two NVIDIA GeForce RTX 4090 GPUs. |
| Software Dependencies | No | All the experiments are conducted using the fairseq implementation of Adafactor 4 and the Hugging Face implementation of Adam... Our code is based on Pytorch package which is standard. |
| Experiment Setup | Yes | Each experiment is conducted over three epochs with a batch size of 128 for BERT-Base/Large and a batch size of 8 for GPT-2. The base learning rate ρ0 is selected via a two-stage grid search. First, we search over the coarse grid {1, 0.1, 0.01, 0.001, 0.0001}. Then, based on the best candidate (e.g., 0.001), we refine the search by evaluating its surrounding values with a step-size equal to one-tenth of the candidate value (e.g., 1 10 4), and choose the best-performing learning rate. All models are trained with the Adafactor optimizer without update clipping, under the parameter setting β2,k = 1 1/kc and ρk = ρ0/kc, where the decay rate c ranges over {0.6, 0.7, 0.8, 0.9, 1.0}. |