Bilevel Optimization under Unbounded Smoothness: A New Algorithm and Convergence Analysis
Authors: Jie Hao, Xiaochuan Gong, Mingrui Liu
ICLR 2024 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Our experiments on hyperrepresentation learning, hyperparameter optimization, and data hyper-cleaning for text classification tasks demonstrate the effectiveness of our proposed algorithm. |
| Researcher Affiliation | Academia | Jie Hao, Xiaochuan Gong, Mingrui Liu Department of Computer Science, George Mason University, Fairfax, VA 22030, USA {jhao6, xgong2, mingruil}@gmu.edu |
| Pseudocode | Yes | Algorithm 1: BO-REP, Algorithm 2: Update Lower, Algorithm 3: Epoch-SGD |
| Open Source Code | Yes | The code is available at https://github.com/Mingrui Liu-ML-Lab/ Bilevel-Optimization-under-Unbounded-Smoothness. |
| Open Datasets | Yes | Hyper-representation experiment is conducted over Amazon Review Dataset (Blitzer et al., 2006), The hyper-cleaning experiments are conducted over the Sentiment140 dataset (Go et al., 2009) for text classification |
| Dataset Splits | Yes | Dval and Dtr denote validation and training sets respectively. |
| Hardware Specification | Yes | To accurately evaluate each algorithm, we use the machine learning framework Py Torch 1.13 to run each algorithm individually on an NVIDIA RTX A6000 graphics card, and record its training and test loss. |
| Software Dependencies | Yes | we use the machine learning framework Py Torch 1.13 |
| Experiment Setup | Yes | We use grid search to tune the lower-level and upper-level step sizes from {0.001, 0.005, 0.01, 0.05, 0.1, 0.5} for all methods. The best combinations of lower-level and upper-level learning rates are (0.05, 0.1) for MAML and ANIL, (0.05, 0.05) for Stoc Bio, (0.1, 0.01) for TTSA, (0.05, 0.05) for SOBA and SABA, (0.05, 0.1) for MA-SOBA, and (0.001, 0.01) for BO-REP. |