Learning a Ground Truth Ranking Using Noisy Approval Votes
Authors: Ioannis Caragiannis, Evi Micha
IJCAI 2017 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We have also conducted experiments with randomized approval (and with a deterministic variant of it) that verify our positive theoretical result; these are also presented in Section 4 together with a discussion on possible extensions of our proof. |
| Researcher Affiliation | Academia | Ioannis Caragiannis University of Patras caragian@ceid.upatras.gr Evi Micha University of Patras michap@ceid.upatras.gr |
| Pseudocode | No | The paper describes procedures and mathematical steps but does not include any formally labeled pseudocode or algorithm blocks. |
| Open Source Code | No | The paper states 'In our experiments, we have used this implementation' referring to an existing method for Mallows, but does not provide concrete access to the source code for their own methodology or experiments. |
| Open Datasets | No | The paper conducts experiments using data generated by the Mallows noise model ('1000 simulations'), rather than utilizing or providing access to a specific publicly available dataset. |
| Dataset Splits | No | The paper describes experimental results from simulations and discusses 'observed accuracy,' but it does not specify explicit train, validation, or test dataset splits. |
| Hardware Specification | No | The paper mentions conducting 'experiments' and 'simulations' but does not provide any specific details about the hardware used, such as GPU/CPU models or memory specifications. |
| Software Dependencies | No | The paper mentions using a specific implementation method ('repeated insertion method') but does not list any specific software components or libraries with their version numbers that would be necessary for reproducibility. |
| Experiment Setup | Yes | We have used profiles with homogeneous Mallows agents with three different parameter values, namely p = 0.95, 0.75, and 0.6. The results are depicted in Figure 1 and show a threshold phenomenon: as the number of agents increases in a short range of values, the accuracy jumps very steeply from 0 to almost 1. The results indicate that randomized approval is considerably better in practice compared to our theoretical bound. For example, in order to achieve accuracy of 90%, profiles with 250, 800, and 2250 agents with Mallows parameters 0.95, 0.75, and 0.6, respectively, are sufficient. The theoretical bounds are approximately 37 000, 120 000, and 750 000. Observed accuracy is measured as the frequency of correctly recovering the ground truth among 1000 simulations. |