Learning to Aggregate Ordinal Labels by Maximizing Separating Width
Authors: Guangyong Chen, Shengyu Zhang, Di Lin, Hui Huang, Pheng Ann Heng
ICML 2017 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Our algorithm is empirically evaluated on several real world datasets, and demonstrates its supremacy over state-of-the-art methods. |
| Researcher Affiliation | Academia | 1The Chinese University of Hong Kong, Hong Kong, China. 2Shenzhen University, China. 3Guangdong Provincial Key Laboratory of Computer Vision and Virtual Reality Technology, Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China. |
| Pseudocode | Yes | Algorithm 1 Our Ordinal Crowdsourcing Method |
| Open Source Code | Yes | the source code with demo can be found on the website1. 1http://appsrv.cse.cuhk.edu.hk/ gychen/ |
| Open Datasets | Yes | We first evaluate our method on three binary benchmark datasets shown in Table 1, include labeling bird species (Welinder et al., 2010) (Bird dataset), recognizing textual entailment (Snow et al., 2008) (RTE dataset) and accessing the relevance of topic-document pairs with a binary judgment in TREC 2011 crowdsourcing track (Gabriella & Matthew, 2011) (TREC dataset). |
| Dataset Splits | No | The paper lists dataset characteristics (e.g., 'Bird 2 108 39 39.0' for classes, items, workers, and labels per item) and mentions 'train' (implicitly by using datasets for training/evaluation) but does not provide specific details on how these datasets were split into training, validation, or test sets (e.g., percentages or counts). |
| Hardware Specification | Yes | all experiments are conducted in a PC with Intel Core i7 1.8GHz CPU and 8.00GB RAM. |
| Software Dependencies | No | The paper does not specify any software dependencies with version numbers (e.g., specific libraries, frameworks, or solvers). |
| Experiment Setup | Yes | For our method, we configure λ1 = λ2 = 1, α = 1M, β = 1N, η = 1 10 5 and initialize zi by the majority voting result. In each run of our method, we generate 80 samples to approximate the posterior distribution and discard the first 10 samples as burn-in steps. |