Obtaining High-Quality Label by Distinguishing between Easy and Hard Items in Crowdsourcing
Authors: Wei Wang, Xiang-Yu Guo, Shao-Yuan Li, Yuan Jiang, Zhi-Hua Zhou
IJCAI 2017 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | The experimental results demonstrate that the proposed approach by learning to distinguish between easy and hard items can significantly improve the label quality. |
| Researcher Affiliation | Academia | Wei Wang, Xiang-Yu Guo, Shao-Yuan Li, Yuan Jiang, and Zhi-Hua Zhou National Key Laboratory for Novel Software Technology Nanjing University, Nanjing 210023, China {wangw,guoxy,lisy,jiangy,zhouzh}@lamda.nju.edu.cn |
| Pseudocode | Yes | Algorithm 1 Difficulty Learning |
| Open Source Code | No | No explicit statement or link providing concrete access to the source code for the methodology described in this paper was found. |
| Open Datasets | No | The paper describes using 'real data' with specific characteristics (1499 images, Fisher vectors, 38 graduate students for labeling), but it does not provide any concrete access information (link, DOI, formal citation with authors/year, or mention of an established benchmark) for this dataset. |
| Dataset Splits | No | The paper states 'We randomly select 20% images to generate the training set L' but does not provide specific information regarding validation splits or further breakdown of the remaining data. |
| Hardware Specification | No | No specific hardware details (like GPU/CPU models, memory, or cloud instance types) used for running experiments were mentioned. |
| Software Dependencies | No | No specific software dependencies with version numbers (e.g., programming languages, libraries, or solvers) were mentioned. |
| Experiment Setup | Yes | The optimization problem in Equation 8 is solved to predict the difficulty of the images in X L, where the kernel function Φ is set to be the linear kernel, α = 10 4 and β = 3. The parameter η is set between 0.1 and 0.3. |