SpiderBoost and Momentum: Faster Variance Reduction Algorithms
Authors: Zhe Wang, Kaiyi Ji, Yi Zhou, Yingbin Liang, Vahid Tarokh
NeurIPS 2019 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | 5 ExperimentsIn this subsection, we compare the performance of SPIDER and Spider Boost for solving the logistic regression problem with a nonconvex regularizer and the nonconvex robust linear regression problem (See Appendix F for the forms of the objective functions). For each problem, we apply two different datasets from the LIBSVM [6]: the a9a dataset (n = 32561, d = 123) and the w8a dataset (n = 49749, d = 300). Figure 1 shows the convergence of the function value gap of both algorithms versus the number of passes that are taken over the data. |
| Researcher Affiliation | Academia | Zhe Wang Department of ECE The Ohio State University wang.10982@osu.edu Kaiyi Ji Department of ECE The Ohio State University ji.367@osu.edu Yi Zhou Department of ECE The University of Utah yi.zhou@utah.edu Yingbin Liang Department of ECE The Ohio State University liang.889@osu.edu Vahid Tarokh Department of ECE Duke University vahid.tarokh@duke.edu |
| Pseudocode | Yes | Algorithm 1 Spider Boost; Algorithm 2 Prox-Spider Boost; Algorithm 3 Prox-Spider Boost-M |
| Open Source Code | No | No explicit statement or link regarding the release of source code for the methodology is provided. |
| Open Datasets | Yes | datasets from the LIBSVM [6]: the a9a dataset (n = 32561, d = 123) and the w8a dataset (n = 49749, d = 300). |
| Dataset Splits | No | The paper does not explicitly provide details about training/validation/test dataset splits. It mentions datasets and epochs but not how the data was partitioned for training, validation, and testing. |
| Hardware Specification | No | The paper does not provide any specific details regarding the hardware used for running the experiments. |
| Software Dependencies | No | The paper does not provide specific software dependencies with version numbers (e.g., programming languages, libraries, frameworks). |
| Experiment Setup | Yes | For both algorithms, we use the same parameter setting except for the stepsize. As specified in [8] for SPIDER, we set η = 0.01 (determined by a prescribed accuracy to guarantee convergence). On the other hand, Spider Boost allows to set η = 0.05. For all algorithms considered, we set their learning rates to be 0.05. For each experiment, we initialize all the algorithms at the same point that is generated randomly from the normal distribution. Also, we choose a fixed mini-batch size 256 and set the epoch length q to be 2n/256 such that all algorithms pass over the entire dataset twice in each epoch. |