Learning Rich Rankings
Authors: Arjun Seshadri, Stephen Ragain, Johan Ugander
NeurIPS 2020 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Our empirical evaluations focus both on predicting out-of-sample rankings as well as predicting sequential entries of rankings as the top entries are revealed. We find that the flexible CRS model we introduce in this work achieves significantly higher out-of-sample likelihood, compared to the PL and Mallows models, across a wide range of applications including ranked choice voting from elections, sushi preferences, Nascar race results, and search engine results. and 5 Empirical results We evaluate the performance of various repeated selection models in learning from and making predictions on empirical datasets, a relative rarity in the theory-focused ranking literature. |
| Researcher Affiliation | Academia | Arjun Seshadri Stanford University aseshadr@stanford.edu Stephen Ragain Stanford University sragain17@gmail.com Johan Ugander Stanford University jugander@stanford.edu |
| Pseudocode | No | The paper describes methods mathematically and textually but does not include any structured pseudocode or algorithm blocks. |
| Open Source Code | Yes | All replication code is publicly available1. 1https://github.com/arjunsesh/lrr-neurips. |
| Open Datasets | Yes | We study four widely studied datasets: the sushi dataset representing ranked food preferences, the dub-n, dub-w, and meath datasets representing ranked choice voting, the nascar dataset representing competitions, and the LETOR collection representing search engine rankings. We provide detailed descriptions of the datasets in Appendix A, as well as an explanation of the more complex PREF-SOC and PREF-SOI collections. and reference [36] Nicholas Mattei and Toby Walsh. Preflib: A library of preference data HTTP://PREFLIB.ORG. In Proceedings of the 3rd International Conference on Algorithmic Decision Theory (ADT 2013), Lecture Notes in Artificial Intelligence. Springer, 2013. |
| Dataset Splits | Yes | For all datasets we use 5-fold cross validation for evaluating test metrics. Using the sushi dataset as an example, for each choice model we train on repeated selection choices for each of 5 folds of the 5,000 rankings in the dataset. |
| Hardware Specification | No | The paper does not provide specific details about the hardware used for running the experiments (e.g., GPU model, CPU model, memory specifications). |
| Software Dependencies | No | The paper mentions 'Pytorch' for implementation and 'Adam' for optimization, but does not provide specific version numbers for these software dependencies. |
| Experiment Setup | Yes | We run Adam with the default parameters (lr = 0.001, β = (0.9, 0.999), ϵ = 1e 8). We use 10 epochs of optimization for the election datasets, where a single epoch converged. |