How Free is Parameter-Free Stochastic Optimization?
Authors: Amit Attia, Tomer Koren
ICML 2024 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Theoretical | We study the problem of parameter-free stochastic optimization, inquiring whether, and under what conditions, do fully parameter-free methods exist: these are methods that achieve convergence rates competitive with optimally tuned methods, without requiring significant knowledge of the true problem parameters. [...] In the non-convex setting, we demonstrate that a simple hyperparameter search technique results in a fully parameter-free method that outperforms more sophisticated state-of-the-art algorithms. We also provide a similar result in the convex setting with access to noisy function values under mild noise assumptions. Finally, assuming only access to stochastic gradients, we establish a lower bound that renders fully parameter-free stochastic convex optimization infeasible, and provide a method which is (partially) parameter-free up to the limit indicated by our lower bound. |
| Researcher Affiliation | Collaboration | 1Blavatnik School of Computer Science, Tel Aviv University 2Google Research Tel Aviv. |
| Pseudocode | Yes | Algorithm 1: Adaptive projected SGD tuning; Algorithm 2: Non-convex SGD tuning; Algorithm 3: Convex SGD tuning |
| Open Source Code | No | The paper does not contain any explicit statement about open-source code availability for the described methodology, nor does it provide a link to a code repository. |
| Open Datasets | No | This is a theoretical paper focused on algorithm design and analysis, and as such, it does not use or provide information about specific datasets for training. |
| Dataset Splits | No | This is a theoretical paper focused on algorithm design and analysis, and therefore does not discuss training, validation, or test splits of datasets. |
| Hardware Specification | No | This is a theoretical paper that focuses on mathematical analysis and algorithm design, and as such, it does not specify any hardware used for experiments. |
| Software Dependencies | No | This is a theoretical paper focused on mathematical analysis and algorithm design; therefore, it does not list specific software dependencies with version numbers. |
| Experiment Setup | No | This is a theoretical paper focused on algorithm design and analysis, and does not provide details of an experimental setup such as hyperparameters or training configurations. |