Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in [1].
A Stochastic Approach to the Subset Selection Problem via Mirror Descent
Authors: Dan Greenstein, Elazar Gershuni, Ilan Ben-Bassat, Yaroslav Fyodorov, Ran Moshe, Fiana Raiber, Alex Shtoff, Oren Somekh, Nadav Hallak
ICLR 2025 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Empirical evaluation of selecting a subset of layers in transfer learning complements our theoretical findings and demonstrates the potential benefits of our approach. We conducted experiments in two setups: (i) Transfer Learning (TL) to showcase a practical application, and (ii) Synthetic experiments for direct evaluation on subset selection tasks. TL experiments are discussed below, with synthetic experiment details in Appendix G. |
| Researcher Affiliation | Collaboration | 1 Technion Israel Institute of Technology, Faculty of Data and Decision Sciences 2 Technion Israel Institute of Technology, Faculty of Computer Science 3 Yahoo Research 4 Technology Innovation Institute 5 Private Consultant |
| Pseudocode | Yes | Algorithm 1: Stochastic Subset Learner |
| Open Source Code | No | The paper does not provide any explicit statement or link regarding the release of source code for the methodology described. |
| Open Datasets | Yes | Concretely, we experiment on TL in vision classification tasks based on two models, Vision Transformer (ViT) Dosovitskiy et al. (2020) and ResNet18 He et al. (2016), and two datasets, CIFAR10 Krizhevsky (2009) and SVHN Netzer et al. (2011). The model weights are pre-trained on ImageNet1K Deng et al. (2009). |
| Dataset Splits | Yes | At each iteration, a subset of k layers is sampled, connected to a trainable fully connected layer, and the resulting model is trained and evaluated on a held-out dataset. ... The model is then evaluated on the calibration set, and we set the loss in Algorithm 1 to 1 calibration accuracy. ... the best layers-subset is evaluated on the validation set to update the HPO process for Algorithm 1. ... evaluate the best in terms of calibration accuracy on the test set. |
| Hardware Specification | Yes | The experiments are carried out on AWS Sagemaker, with the instance types ml.g4dn.16xlarge , with 64 v CPU, 1 Nvidia t4 tensor core GPU, and an Intel Xeon Family physical processor. |
| Software Dependencies | No | The paper does not provide any specific software names with version numbers. |
| Experiment Setup | Yes | learning rate in [0.001, 0.1] and batch size in {32, 64, 128}. ... The model is then evaluated on the calibration set, and we set the loss in Algorithm 1 to 1 calibration accuracy. We update the layer weights and repeat for T iterations according to Algorithm 1 (T = 50). |