Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in [1].
Computational and Statistical Tradeoffs in Learning to Rank
Authors: Ashish Khetan, Sewoong Oh
NeurIPS 2016 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We provide numerical experiments confirming the theoretical guarantees. Figure 2: The time-data trade-off for fixed accuracy (left) and accuracy improvement for increased computation M (middle). Generalized Rank-Breaking (GRB) achieves the oracle lower bound and significantly improves upon Pairwise Rank-Breaking (PRB) (right). Figure 3: Accuracy degrades as (κ m) gets small and as the dynamic range b gets large. In Figure 3 left and middle panel, we compare performance of our algorithm with pairwise breaking, Cramer Rao lower bound and oracle MLE lower bound. On sushi preferences [14] and jester dataset [11], we improve over pairwise breaking and achieves same performance as the oracle MLE. |
| Researcher Affiliation | Academia | Ashish Khetan and Sewoong Oh Department of ISE, University of Illinois at Urbana-Champaign Email: EMAIL |
| Pseudocode | No | The paper describes the proposed method but does not include any structured pseudocode or algorithm blocks. |
| Open Source Code | No | The paper does not provide any concrete access to source code (specific repository link, explicit code release statement, or code in supplementary materials) for the methodology described in this paper. |
| Open Datasets | Yes | On sushi preferences [14] and jester dataset [11], we improve over pairwise breaking and achieves same performance as the oracle MLE. Full rankings over κ = 10 types of sushi are randomly chosen from d = 100 types of sushi are provided by n = 5000 individuals. |
| Dataset Splits | No | The paper does not provide specific dataset split information (exact percentages, sample counts, citations to predefined splits, or detailed splitting methodology) needed to reproduce the data partitioning into train/validation/test sets. |
| Hardware Specification | No | The paper does not provide specific hardware details (exact GPU/CPU models, processor types with speeds, memory amounts, or detailed computer specifications) used for running its experiments. |
| Software Dependencies | No | The paper does not provide specific ancillary software details (e.g., library or solver names with version numbers) needed to replicate the experiment. |
| Experiment Setup | Yes | We fix d = 256, ℓ= 5, ma = a for a {1, 2, 3, 4, 5}, and sample posets from the canonical scenario, except that each user is presented κ = 32 random items. The PL weights are chosen i.i.d. U[ 2, 2]. We fix d = 512, n = 105, θ chosen i.i.d. uniformly over [ 2, 2]. |