Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in [1].
Stochastically Transitive Models for Pairwise Comparisons: Statistical and Computational Issues
Authors: Nihar Shah, Sivaraman Balakrishnan, Aditya Guntuboyina, Martin Wainwright
ICML 2016 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We complement our theoretical results with thorough numerical simulations. The simulations in this section add to the simulation results of Section 2.4 (Figure 1) demonstrating a large class of matrices in the SST class that cannot be represented by any parametric class. We investigate the performance of the soft-SVT estimator (Section 3.2) and the maximum likelihood estimator under the Thurstone model (Section 2.3). Figure 2 depicts the results of the simulations based on observations of the entire matrix Y. |
| Researcher Affiliation | Academia | Nihar B. Shah EMAIL Sivaraman Balakrishnan] EMAIL Adityanand Guntuboyina EMAIL Martin J. Wainwright EMAIL Dept. of EECS, Dept. of Statistics, University of California, Berkeley ] Dept. of Statistics, Carnegie Mellon University |
| Pseudocode | No | The paper describes algorithms (e.g., SVT estimator, two-step estimator) in prose, detailing steps and mathematical formulations, but it does not include any dedicated pseudocode blocks, figures, or algorithm listings. |
| Open Source Code | No | The paper does not contain any explicit statement about releasing source code for the methodology or a link to a code repository. |
| Open Datasets | No | In our simulations, we generate the ground truth M in the following five ways: Uniform, Thurstone, Bradley-Terry-Luce (BTL), High SNR, Independent bands. |
| Dataset Splits | No | The paper describes how the ground truth data (M) for simulations is generated and then how estimators are applied to observations (Y) from this M. However, it does not specify any training, validation, or test dataset splits for its own experimental setup or model training beyond stating that observations are made of the entire matrix Y. |
| Hardware Specification | No | The paper does not provide any specific details about the hardware (e.g., GPU models, CPU types, memory) used to run the simulations or experiments. |
| Software Dependencies | No | The paper mentions statistical models and algorithms but does not specify any software names with version numbers (e.g., Python 3.x, PyTorch 1.x, specific solvers) that would be needed for reproducibility. |
| Experiment Setup | No | The paper describes the data generation process for simulations and the estimators used (SVT, Thurstone MLE), including a threshold parameter for SVT. However, it lacks concrete details typical of an experimental setup, such as learning rates, batch sizes, specific optimizers, number of training epochs, or other hyperparameter values for the algorithms when applied in the simulations. It describes the algorithms themselves rather than their specific experimental configurations. |