Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in Coakley et alK. L. Coakley, T. Snelleman, H. Hoos, and O. E. Gundersen, "The embrace of open science: An analysis of a decade of AI research and 56 800 conference papers," Under Review, 2026..
PaZO: Preconditioned Accelerated Zeroth-Order Optimization for Fine-Tuning LLMs
Authors: Hanzhen Zhao, Ding Shihong, Cong Fang, Zhouchen Lin
NeurIPS 2025 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Extensive experiments on diverse downstream tasks with models like Ro BERTa-large and OPT show Pa ZO s effectiveness. Compared to other zerothorder baselines, Pa ZO achieves better performance across models and tasks. |
| Researcher Affiliation | Academia | 1 State Key Lab of General AI, School of Intelligence Science and Technology, Peking University 2 Institute for Artificial Intelligence, Peking University 3 Pazhou Laboratory (Huangpu), Guangzhou, Guangdong, China |
| Pseudocode | Yes | Algorithm 1 Pa ZO (Practical Form) |
| Open Source Code | Yes | Code is available at Code. |
| Open Datasets | Yes | For Ro BERTa-large, we consider classification datasets: SST-2 [47], SST-5 [47], TREC [51], MNLI [54], SNLI [7], and RTE [20, 14, 18, 6]. ... For OPT-1.3B, we consider the Super GLUE dataset collection [52], including: Bool Q [13], CB [15], COPA [45], Multi RC [27], Re Co RD [56], RTE [20, 14, 18, 6], Wi C [43], and WSC [30]. |
| Dataset Splits | Yes | We sample k examples per class for k = 16, running zerothshot learning, LP, fine-tuning, Me ZO and Pa ZO. ... For training and validation, we set k = 16, which means that we have 16 examples per class for both training and validation. ... We randomly sample 1000, 500, and 1000 examples for training, validation, and test sets, respectively, for each dataset. |
| Hardware Specification | Yes | All results are measured on the same dataset (SST-2) and GPUs (24GB 3090), with each result averaged over 100 steps. ... Table 5: Peak memory on the Multi RC (average tokens=400) dataset. Method zero-shot/Me ZO Pa ZO ICL FT FT (prefix) 1.3B 1x A6000 (4GB) 1x A6000 (9GB) 1x A6000 (6GB) 1x A6000 (27GB) 1x A6000 (19GB) 2.7B 1x A6000 (7GB) 1x A6000 (14GB) 1x A6000 (8GB) 2x A6000 (55GB) 1x A6000 (29GB) 6.7B 1x A6000 (14GB) 1x A6000 (30GB) 1x A6000 (16GB) 4x A6000 (156GB) 1x A6000 (46GB) 13B 1x A6000 (26GB) 2x A6000 (54GB) 1x A6000 (29GB) 8x A6000 (316GB) 4x A6000 (158GB) |
| Software Dependencies | No | The paper mentions models like Ro BERTa-large and OPT-1.3B, and optimizers like Adam. It does not explicitly mention software versions like Python, PyTorch, or CUDA with specific version numbers. |
| Experiment Setup | Yes | We use the hyperparameters in Table 3 for experiments on Ro BERTa-large. Previous work [37] shows that the choice of ϵ seems to not significantly impact the performance, and using a larger batch size consistently yielded faster optimization. We use the hyperparameters in Table 4 for zeroth-order methods on OPT-1.3B. We use linear learning scheduling for first-order fine-tuning methods with backpropagation, and constant learning rate for all zeroth-order methods. |