Aggregating Ordinal Labels from Crowds by Minimax Conditional Entropy
Authors: Dengyong Zhou, Qiang Liu, John Platt, Christopher Meek
ICML 2014 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Empirical evaluations on real datasets demonstrate significant improvements over existing methods.In this section, we report empirical results of our method on real crowdsourced data in comparison with state-of-the-art methods that aggregate multiclass or ordinal labels. |
| Researcher Affiliation | Collaboration | Dengyong Zhou DENZHO@MICROSOFT.COM Microsoft Research, Redmond, WA 98052 Qiang Liu QLIU1@UCI.EDU University of California, Irvine, CA 92697 John C. Platt JPLATT@MICROSOFT.COM Microsoft Research, Redmond, WA 98052 Christopher Meek MEEK@MICROSOFT.COM Microsoft Research, Redmond, WA 98052 |
| Pseudocode | Yes | Algorithm 1 Regularized Minimax Conditional Entropy |
| Open Source Code | No | The paper mentions using 'an open source implementation of latent trait analysis by Mineiro (2011)' for comparison but does not state that the source code for the methodology described in this paper is publicly available. |
| Open Datasets | Yes | The web search relevance rating dataset contains 2665 query-URL pairs and 177 workers (Zhou et al., 2012).The price dataset consists of 80 household items collected from stores such as Amazon and Costco. The prices of those items are estimated by 155 undergraduate students (Liu et al., 2013). |
| Dataset Splits | Yes | If the true labels of a subset of items are known such subsets are usually referred to as validation sets, we may choose (α, β) such that those known true labels can be best predicted.We randomly select 1500 pairs to form a test set, and then select 10 to 100 pairs from the remaining pairs to form validation sets. |
| Hardware Specification | No | The paper does not provide specific hardware details (e.g., exact GPU/CPU models, processor types, memory amounts, or detailed computer specifications) used for running its experiments. |
| Software Dependencies | No | The paper mentions using 'an open source implementation of latent trait analysis by Mineiro (2011)' but does not provide specific version numbers for this or any other software dependencies crucial for replicating the experiments. |
| Experiment Setup | Yes | The cross-validation parameter k is typically set to 5 or 10.In our experiments, we choose γ from {1/4, 1/2, 1, 2, 4}.α = γ (number of classes)2, β = number of labels per worker / number of labels per item α. |