Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in [1].
Simultaneous Preference and Metric Learning from Paired Comparisons
Authors: Austin Xu, Mark Davenport
NeurIPS 2020 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We conduct extensive experiments on synthetic and real-world datasets to exhibit the effectiveness of our algorithm. We demonstrate the effectiveness of our approach through experiments on both synthetic and real-world datasets. |
| Researcher Affiliation | Academia | Austin Xu Georgia Institute of Technology Atlanta, GA 30332 EMAIL Mark Davenport Georgia Institute of Technology Atlanta, GA 30332 EMAIL |
| Pseudocode | No | The paper describes the estimation strategy and optimization problems mathematically, but does not include any explicit pseudocode or algorithm blocks. |
| Open Source Code | Yes | Code available at https://github.com/austinxu87/Ideal Point Metric |
| Open Datasets | No | The paper uses "synthetic and real-world datasets" including two internal "Ph D program admissions datasets from Georgia Tech School of Electrical and Computer Engineering" (Unranked Candidates and Ranked Candidates datasets). No concrete access information (link, DOI, formal citation) is provided for these datasets, nor are they described as publicly available. |
| Dataset Splits | No | The paper does not explicitly provide specific training, validation, or test dataset splits (e.g., percentages or sample counts) for its experiments. While it discusses general concepts like 'validation', it does not specify how its own datasets were partitioned. |
| Hardware Specification | No | The paper does not provide any specific hardware details such as GPU or CPU models, memory, or cloud instance types used for running the experiments. |
| Software Dependencies | Yes | The above formulation is a convex (semi-definite) program and can be solved by standard tools such as CVX [39, 40]. |
| Experiment Setup | Yes | Regularization parameters: γ1 = 2, γ2 = 0.002, γ3 = 0.001, = 1. Regularization parameters: γ(0)1 = 2, γ(0)2 = 0.002, γ(0)3 = 0.0001, (0) = 1; γ(k)1 = 2 3, γ(k)3 = 7 1500, (k) = 1 |