Stochastic Optimization with Heavy-Tailed Noise via Accelerated Gradient Clipping
Authors: Eduard Gorbunov, Marina Danilova, Alexander Gasnikov
NeurIPS 2020 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We conduct several numerical experiments with the proposed methods in order to justify the theory we develop. In particular, we show that clipped-SSTM can outperform SGD and clipped-SGD in practice even without using large batchsizes. |
| Researcher Affiliation | Academia | Eduard Gorbunov MIPT and HSE, Russia Marina Danilova ICS RAS and MIPT, Russia Alexander Gasnikov MIPT and HSE, Russia |
| Pseudocode | Yes | Algorithm 1 Clipped Stochastic Similar Triangles Method (clipped-SSTM) |
| Open Source Code | Yes | One can find the code here: https://github.com/eduardgorbunov/accelerated_clipping. |
| Open Datasets | Yes | We have tested4 clipped-SSTM and clipped-SGD on the logistic regression problem, the datasets were taken from LIBSVM library [4]. |
| Dataset Splits | No | The paper mentions using datasets from LIBSVM library but does not explicitly state the training, validation, or test dataset splits. |
| Hardware Specification | No | The paper does not provide specific hardware details (e.g., GPU/CPU models, memory amounts) used for running its experiments. |
| Software Dependencies | No | To implement methods we use Python 3.7 and standard libraries. |
| Experiment Setup | Yes | For all methods we used constant batchsizes m, stepsizes and clipping levels were tuned, see Section H.2 for the details. |