RUMs from Head-to-Head Contests
Authors: Matteo Almanza, Flavio Chierichetti, Ravi Kumar, Alessandro Panconesi, Andrew Tomkins
ICML 2022 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We present a comprehensive set of experiments comparing the RUM algorithm to other standard and recent approaches to pairwise choice. ... In empirical evaluations we observe that RUMRUNNER is neither subject to poor local optima nor has stability issues. |
| Researcher Affiliation | Collaboration | 1Algorand Labs 2This work was carried out while the author was at Sapienza University of Rome. 3Sapienza University of Rome 4Google Mountain View. |
| Pseudocode | Yes | Algorithm 1 RUMRUNNER, a heuristic for Problem 6., Algorithm 2 A randomized local search for Viol-HP, i.e., the separation oracle. |
| Open Source Code | No | The paper does not provide a link to its own source code or state that it is open-source. It only mentions the use of a third-party solver (IBM CPLEX) with a link to its website. |
| Open Datasets | Yes | We measured the performances of the algorithms on a number of datasets: a class of commonly studied election datasets A5, A9, A17, A48, A81 (Tideman, 2006), one videogame dataset representing matchups between Super Street Fighter IV (SF) characters (Chen & Joachims, 2016), and the Jester rating dataset (Goldberg et al., 2001). |
| Dataset Splits | Yes | Our tests follows those of Makhijani & Ugander (2019): we use a five-fold cross-validation, and measure their RMSE as a metric. ... We then split, uniformly at random, the list into five parts, conditioned on each part having length equal to either t/5 or t/5 . |
| Hardware Specification | Yes | All experiments were done on commodity hardware10. 1012-Core Ryzen 3900X with 64GB of RAM |
| Software Dependencies | No | The paper mentions "IBM cplex9" but does not provide a specific version number (e.g., "CPLEX 9.0") nor versions for other software components like Python. |
| Experiment Setup | Yes | In experiments, we set t = 100. ... stopping an iteration if the gain in the last step was less than 10 5. ... We randomly sampled 10 splittings obtaining, for each dataset, 50 train/test pairs. |