Iterative Hard Thresholding with Adaptive Regularization: Sparser Solutions Without Sacrificing Runtime
Authors: Kyriakos Axiotis, Maxim Sviridenko
ICML 2022 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | In this section we present numerical experiments in order to compare the performance of IHT and regularized IHT (Algorithm 1) in training sparse linear models. |
| Researcher Affiliation | Collaboration | 1MIT 2Yahoo! Research. |
| Pseudocode | Yes | Algorithm 1 Regularized IHT |
| Open Source Code | Yes | In Figure 2 we have python implementations of the IHT and regularized IHT algorithms that we use for our experiments. As can be seen, both implementations are pretty short. |
| Open Datasets | Yes | We first experiment with real data, specifically the year regression dataset from UCI (Dua & Graff, 2017) and the rcv1 binary classification dataset (Lewis et al., 2004), which have been previously used in the literature. |
| Dataset Splits | No | The paper describes using real and synthetic data for experiments and preprocessing steps, but does not provide specific details on train/validation/test dataset splits (e.g., percentages, sample counts, or explicit mention of validation sets). |
| Hardware Specification | Yes | All the experiments were run on a single 2.6GHz Intel Core i7 core of a 2019 Mac Book Pro with 16GB DDR4 RAM using Python 3.9.10. |
| Software Dependencies | Yes | All the experiments were run on a single 2.6GHz Intel Core i7 core of a 2019 Mac Book Pro with 16GB DDR4 RAM using Python 3.9.10. |
| Experiment Setup | Yes | For our experiments we use ρ = 0.1. For the weight step size of regularized IHT, we set the weight step size to c = s /T... for each algorithm we pick the best fixed step size of the form 2i/s for integer i 0, where s is the fixed sparsity level. The best step sizes of IHT and regularized IHT end up being 2/s, 4/s respectively for the linear regression example, and 8/s, 16/s respectively for the logistic regression example. |