Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in [1].
Aggregating Ordinal Labels from Crowds by Minimax Conditional Entropy
Authors: Dengyong Zhou, Qiang Liu, John Platt, Christopher Meek
ICML 2014 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Empirical evaluations on real datasets demonstrate significant improvements over existing methods.In this section, we report empirical results of our method on real crowdsourced data in comparison with state-of-the-art methods that aggregate multiclass or ordinal labels. |
| Researcher Affiliation | Collaboration | Dengyong Zhou EMAIL Microsoft Research, Redmond, WA 98052 Qiang Liu EMAIL University of California, Irvine, CA 92697 John C. Platt EMAIL Microsoft Research, Redmond, WA 98052 Christopher Meek EMAIL Microsoft Research, Redmond, WA 98052 |
| Pseudocode | Yes | Algorithm 1 Regularized Minimax Conditional Entropy |
| Open Source Code | No | The paper mentions using 'an open source implementation of latent trait analysis by Mineiro (2011)' for comparison but does not state that the source code for the methodology described in this paper is publicly available. |
| Open Datasets | Yes | The web search relevance rating dataset contains 2665 query-URL pairs and 177 workers (Zhou et al., 2012).The price dataset consists of 80 household items collected from stores such as Amazon and Costco. The prices of those items are estimated by 155 undergraduate students (Liu et al., 2013). |
| Dataset Splits | Yes | If the true labels of a subset of items are known such subsets are usually referred to as validation sets, we may choose (α, β) such that those known true labels can be best predicted.We randomly select 1500 pairs to form a test set, and then select 10 to 100 pairs from the remaining pairs to form validation sets. |
| Hardware Specification | No | The paper does not provide specific hardware details (e.g., exact GPU/CPU models, processor types, memory amounts, or detailed computer specifications) used for running its experiments. |
| Software Dependencies | No | The paper mentions using 'an open source implementation of latent trait analysis by Mineiro (2011)' but does not provide specific version numbers for this or any other software dependencies crucial for replicating the experiments. |
| Experiment Setup | Yes | The cross-validation parameter k is typically set to 5 or 10.In our experiments, we choose γ from {1/4, 1/2, 1, 2, 4}.α = γ (number of classes)2, β = number of labels per worker / number of labels per item α. |