Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in Coakley et alK. L. Coakley, T. Snelleman, H. Hoos, and O. E. Gundersen, "The embrace of open science: An analysis of a decade of AI research and 56 800 conference papers," Under Review, 2026..
How Hard Is the Manipulative Design of Scoring Systems?
Authors: Dorothea Baumeister, Tobias Hogrebe
IJCAI 2019 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | In addition to the theoretical results, we also provide a case study on Formula 1 data. For all seasons from 1961 to 2008, we checked whether there exist scoring systems that result in a different winner. |
| Researcher Affiliation | Academia | Dorothea Baumeister and Tobias Hogrebe Institut f ur Informatik, Heinrich-Heine-Universit at D usseldorf |
| Pseudocode | No | The paper includes mathematical formulations for Linear Programs (LPs) and Integer Linear Programs (ILPs), but it does not present any pseudocode or clearly labeled algorithm blocks. |
| Open Source Code | No | The paper does not provide any explicit statements about releasing source code or links to a code repository for the methodology described. |
| Open Datasets | Yes | We consider real-life competition data from the Formula 1 from the seasons 1961 to 2008. To be precise, we use the version of the data available at Pref Lib.org [Mattei and Walsh, 2013]. |
| Dataset Splits | No | The paper states, 'each instance contains exactly one pro๏ฌle' and analyzes data from different seasons. It does not describe typical training/validation/test splits for machine learning experiments, as it's an analysis of existing data instances. |
| Hardware Specification | No | The paper does not provide any specific details about the hardware used to run the experiments, such as CPU or GPU models, or memory specifications. It only mentions 'on a standard machine'. |
| Software Dependencies | No | The paper states, 'The ILP was then solved using CPLEX,' but it does not specify the version number of CPLEX or any other software dependencies with their versions. |
| Experiment Setup | Yes | For the exact solution, we have implemented the LP from the proof of Theorem 1 as an ILP with ฯต = 1 including the D1 formulation from Theorem 5 and additionally added requirement (I.), (II.), or (III.). |