Faster Approximation of Probabilistic and Distributional Values via Least Squares
Authors: Weida Li, Yaoliang Yu
ICLR 2024 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Our experiments verify the faster convergence of GELS, and demonstrate the effectiveness of Tr ELS in learning distributional values. |
| Researcher Affiliation | Academia | Weida Li vidaslee@gmail.com Yaoliang Yu School of Computer Science University of Waterloo Vector Institute yaoliang.yu@uwaterloo.ca |
| Pseudocode | Yes | Algorithm 1: GELS-R (Generic Estimator based on Least Squares for Rankings) Input: A dataset Dtr [n] to be valuated, a utility function U n Gn, a weight vector q Rn 1 defined by qs = n s (pn s + pn s+1), and a total number T of samples Output: An unbiased estimate ˆr to R(U n) up to some scalar |
| Open Source Code | Yes | Our code is available at https://github.com/watml/fastpvalue. |
| Open Datasets | Yes | The classification datasets employed are from open resources, which are iris, wind (both are from Open ML), FMNIST (Xiao et al. 2017) and MNIST. |
| Dataset Splits | Yes | For all utility functions, their outputs are set to be the performance of the trained models reported on a dataset Dperf disjoint from Dtr. Precisely, U n(S) reports the performance of the specified model trained on S Dtr [n]... To evaluate the specified probabilistic values exactly, we set |Dtr| = |Dperf| = 24... For training estimators, we employ the Adam optimizer (Kingma and Ba 2014) with a learning rate 0.001. The batch size Z is set to be 10, 000, and we randomly generate 1, 000 batches... Moreover, 200 data, denoted by Dval, are taken from the training dataset for selecting the best trained models. For the results in the second row of Figure 2, we report on another 200 data extracted from the test dataset, for which we refer to as Dtest. To sum, Dtr, Dperf, Dval and Dtest are all disjoint, which means Dval and Dtest are composed of unseen data. |
| Hardware Specification | No | The paper does not provide specific hardware details (e.g., CPU/GPU models, memory) used for running the experiments. It only mentions the optimizers and models used. |
| Software Dependencies | No | The paper mentions software components like 'Logistic regression', 'Le Net', 'SGD optimizer', and 'Adam optimizer' but does not specify their version numbers or any other key software dependencies with version information required for reproducibility. |
| Experiment Setup | Yes | The SGD optimizer with a learning rate 1.0 is employed for iris and wind, whereas we set the learning rate to be 0.1 instead for MNIST and FMNIST. To be computationally efficient, we adopt one-mini-batch one-epoch learning for evaluating utility functions (Ghorbani and Zou 2019). For training estimators, we employ the Adam optimizer (Kingma and Ba 2014) with a learning rate 0.001. The batch size Z is set to be 10, 000, and we randomly generate 1, 000 batches. |