Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in [1].
Rank Aggregation Using Scoring Rules
Authors: Niclas Boehmer, Robert Bredereck, Dominik Peters
AAAI 2023 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | To understand how and whether the three families of methods practically differ from each other, and how they relate to Kemeny s method, we perform extensive simulations based on synthetic data (sampled using the Mallows and Euclidean models). |
| Researcher Affiliation | Academia | 1 Algorithmics and Computational Complexity, Technische Universit at Berlin 2 Institut f ur Informatik, TU Clausthal 3 CNRS, LAMSADE, Universit e Paris Dauphine PSL |
| Pseudocode | No | The paper provides definitions and describes algorithms textually (e.g., 'Deļ¬nition 3.2 (Sequential-s-Winner; Seq.-s-Winner). Let s be a scoring system. The social preference function Seq.-s-Winner is deļ¬ned recursively as follows'), but it does not include formal pseudocode blocks or labeled algorithm figures. |
| Open Source Code | Yes | The code of our experiments is available at github.com/n-boehmer/Rank-Aggregation. |
| Open Datasets | No | The paper states, 'We conduct simulations on proļ¬les generated using the Mallows model (Mallows 1957) (as observed by Boehmer et al. (2021) real-world proļ¬les often seem to be close to some Mallows proļ¬le).' While the Mallows model is known, the authors generate their own synthetic data and do not provide access to the specific generated datasets used in their experiments via a link or citation to a public repository. |
| Dataset Splits | No | The paper states 'we sampled 10 000 proļ¬les for each norm-Ļ {0, 0.1, . . . , 0.9, 1}' but does not specify any training, validation, or test dataset splits. The entire set of sampled profiles appears to be used for analysis rather than being partitioned into distinct splits. |
| Hardware Specification | No | The paper describes conducting 'extensive simulations' but does not provide any specific details about the hardware (e.g., CPU, GPU models, memory, or cloud resources) used to run these experiments. |
| Software Dependencies | No | The paper does not list specific software components with their version numbers (e.g., programming languages, libraries, or solvers) that would be needed to reproduce the experiments. |
| Experiment Setup | Yes | To deal with ties in the computation of our rules, each time we sample a ranking proļ¬le over candidates C, we also sample a ranking tie L(C) uniformly at random and break ties according to tie for all rules. We conduct simulations on proļ¬les generated using the Mallows model (Mallows 1957). We sampled 10 000 proļ¬les for each norm-Ļ {0, 0.1, . . . , 0.9, 1}. |