Aggregating Binary Judgments Ranked by Accuracy
Authors: Daniel Halpern, Gregory Kehne, Dominik Peters, Ariel D. Procaccia, Nisarg Shah, Piotr Skowron5456-5463
AAAI 2021 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We show that our algorithms perform well empirically using real and synthetic data in collaborative filtering and political prediction domains. |
| Researcher Affiliation | Academia | Daniel Halpern,1 Gregory Kehne,2 Dominik Peters,1 Ariel D. Procaccia,1 Nisarg Shah,3 Piotr Skowron4 1 Harvard University, 2 Carnegie Mellon University, 3 University of Toronto, 4 University of Warsaw |
| Pseudocode | Yes | Algorithm 1: OPT-LIKELIHOOD Input: Judgment profile X {0, 1}n, y {0, 1} Output: Optimistic likelihood L [X; G = y] if n = 1 then return 1I[X1=y] (1/2)I[X1 =y] end [Find the prefix of X with the highest density of y] i index maximizing (1/i) Pi j=1 I[Xj = y], breaking ties in favor of larger indices d (1/i) Pi j=1 I[Xj = y] r max {d, 1/2} L OPT-LIKELIHOOD((Xi+1, . . . , Xn), y) return rd(1 r)1 d i L |
| Open Source Code | No | The paper mentions a third-party library, 'fancyimpute python library', and provides a link to its GitHub repository. However, it does not provide concrete access (specific repository link or explicit statement) to the source code for the methodology described in *this* paper. |
| Open Datasets | Yes | We evaluate the rules on two datasets from the Pref Lib library (Mattei and Walsh 2013), and on a synthetic dataset. ... We use a dataset from Five Thirty Eight of polling data from the 2016 US Presidential Election. |
| Dataset Splits | No | The paper mentions 'We hide an h fraction of randomly selected entries in the matrix (h {0.5, 0.8, 0.9})' for some datasets, and describes setting up 'ground truth' for political predictions. However, it does not provide specific training/validation/test dataset splits (exact percentages, sample counts, or citations to predefined splits) needed to reproduce the data partitioning for model training or evaluation, beyond indicating that some data is hidden for testing. |
| Hardware Specification | No | The paper discusses running 'computer simulations' but does not provide specific hardware details (exact GPU/CPU models, processor types, or memory amounts) used for its experiments. |
| Software Dependencies | No | The paper mentions using the 'fancyimpute python library' but does not provide specific version numbers for this or any other software dependencies needed to replicate the experiment. |
| Experiment Setup | No | The paper describes some aspects of the experimental setup, such as how similarity scores are calculated or how data is truncated, and how political data is prepared. However, it does not provide specific experimental setup details like concrete hyperparameter values, training configurations, or optimizer settings for their proposed algorithms. |