Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in Coakley et alK. L. Coakley, T. Snelleman, H. Hoos, and O. E. Gundersen, "The embrace of open science: An analysis of a decade of AI research and 56 800 conference papers," Under Review, 2026..
Rank Aggregation Using Scoring Rules
Authors: Niclas Boehmer, Robert Bredereck, Dominik Peters
AAAI 2023 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | To understand how and whether the three families of methods practically differ from each other, and how they relate to Kemeny s method, we perform extensive simulations based on synthetic data (sampled using the Mallows and Euclidean models). |
| Researcher Affiliation | Academia | 1 Algorithmics and Computational Complexity, Technische Universit at Berlin 2 Institut f ur Informatik, TU Clausthal 3 CNRS, LAMSADE, Universit e Paris Dauphine PSL |
| Pseudocode | No | The paper provides definitions and describes algorithms textually (e.g., 'Deļ¬nition 3.2 (Sequential-s-Winner; Seq.-s-Winner). Let s be a scoring system. The social preference function Seq.-s-Winner is deļ¬ned recursively as follows'), but it does not include formal pseudocode blocks or labeled algorithm figures. |
| Open Source Code | Yes | The code of our experiments is available at github.com/n-boehmer/Rank-Aggregation. |
| Open Datasets | No | The paper states, 'We conduct simulations on proļ¬les generated using the Mallows model (Mallows 1957) (as observed by Boehmer et al. (2021) real-world proļ¬les often seem to be close to some Mallows proļ¬le).' While the Mallows model is known, the authors generate their own synthetic data and do not provide access to the specific generated datasets used in their experiments via a link or citation to a public repository. |
| Dataset Splits | No | The paper states 'we sampled 10 000 proļ¬les for each norm-Ļ {0, 0.1, . . . , 0.9, 1}' but does not specify any training, validation, or test dataset splits. The entire set of sampled profiles appears to be used for analysis rather than being partitioned into distinct splits. |
| Hardware Specification | No | The paper describes conducting 'extensive simulations' but does not provide any specific details about the hardware (e.g., CPU, GPU models, memory, or cloud resources) used to run these experiments. |
| Software Dependencies | No | The paper does not list specific software components with their version numbers (e.g., programming languages, libraries, or solvers) that would be needed to reproduce the experiments. |
| Experiment Setup | Yes | To deal with ties in the computation of our rules, each time we sample a ranking proļ¬le over candidates C, we also sample a ranking tie L(C) uniformly at random and break ties according to tie for all rules. We conduct simulations on proļ¬les generated using the Mallows model (Mallows 1957). We sampled 10 000 proļ¬les for each norm-Ļ {0, 0.1, . . . , 0.9, 1}. |