Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in Coakley et alK. L. Coakley, T. Snelleman, H. Hoos, and O. E. Gundersen, "The embrace of open science: An analysis of a decade of AI research and 56 800 conference papers," Under Review, 2026..
Zeroth-Order Optimization Finds Flat Minima
Authors: Liang Zhang, Bingcong Li, Kiran K Thekumparampil, Sewoong Oh, Michael Muehlebach, Niao He
NeurIPS 2025 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We provide empirical evaluations to examine the behavior of the trace of Hessian under zerothorder optimization across three settings: a test function (Figure 1), binary classification tasks using overparameterized SVMs and logistic regression (Figure 2), and language model fine-tuning tasks with Ro BERTa [61] (Figure 3). |
| Researcher Affiliation | Collaboration | Liang Zhang ETH Zurich & Max Planck Institute EMAIL; Bingcong Li ETH Zurich EMAIL; Kiran Koshy Thekumparampil Amazon EMAIL; Sewoong Oh University of Washington EMAIL; Michael Muehlebach Max Planck Institute EMAIL; Niao He ETH Zurich EMAIL |
| Pseudocode | Yes | Algorithm 1 Zeroth-Order Optimization with the Two-Point Estimator |
| Open Source Code | Yes | Our code is available at https://github.com/Liang137/Flat Zero. |
| Open Datasets | Yes | We use two standard binary classification benchmarks from the LIBSVM library [14]: a5a (N = 6, 414, d = 123) and w5a (N = 9, 888, d = 300) [76]... SST-2 and SST-5 [82] for sentiment classification, and TREC [84] for topic classification. All experiments are tested on a single NVIDIA H100 GPU with 80 Gi B memory. |
| Dataset Splits | Yes | The a5a dataset contains N = 6, 414 training and 26, 147 test samples with d = 123 features, and the w5a dataset contains N = 9, 888 training and 39, 861 test samples with d = 300 features... The training set is constructed by sampling K examples per class from the original dataset, and the test set is built by randomly selecting 1, 000 examples from the original test dataset. |
| Hardware Specification | Yes | All experiments are tested on a single NVIDIA H100 GPU with 80 Gi B memory... Each run is executed on a CPU and takes approximately 1 second. |
| Software Dependencies | No | The paper mentions Ro BERTa-Large (355M parameters) [61] and LIBSVM library [14] but does not specify version numbers for these or other software dependencies like programming languages or deep learning frameworks. |
| Experiment Setup | Yes | Gradient descent uses a stepsize of 0.01, and zeroth-order optimization uses 0.001... For SVMs, the stepsize is 0.0001, and the smoothing parameter λ = 0.05. For logistic regression on a5a, the stepsize is 0.01, and the smoothing parameter λ = 0.1... We fix the number of iterations to 100,000 for K = 32 and 50,000 for K = 256... The stepsize for gradient descent is set to 5e-5 for all cases. Zeroth-order optimization uses a stepsize of 1e-5 for SST-2, and 5e-6 for SST-5 and TREC. The smoothing parameter λ in zeroth-order optimization is set to 2e-3 for SST-2 and TREC, and 1e-3 for SST-5. |