Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in [1].
Obtaining High-Quality Label by Distinguishing between Easy and Hard Items in Crowdsourcing
Authors: Wei Wang, Xiang-Yu Guo, Shao-Yuan Li, Yuan Jiang, Zhi-Hua Zhou
IJCAI 2017 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | The experimental results demonstrate that the proposed approach by learning to distinguish between easy and hard items can significantly improve the label quality. |
| Researcher Affiliation | Academia | Wei Wang, Xiang-Yu Guo, Shao-Yuan Li, Yuan Jiang, and Zhi-Hua Zhou National Key Laboratory for Novel Software Technology Nanjing University, Nanjing 210023, China EMAIL |
| Pseudocode | Yes | Algorithm 1 Difficulty Learning |
| Open Source Code | No | No explicit statement or link providing concrete access to the source code for the methodology described in this paper was found. |
| Open Datasets | No | The paper describes using 'real data' with specific characteristics (1499 images, Fisher vectors, 38 graduate students for labeling), but it does not provide any concrete access information (link, DOI, formal citation with authors/year, or mention of an established benchmark) for this dataset. |
| Dataset Splits | No | The paper states 'We randomly select 20% images to generate the training set L' but does not provide specific information regarding validation splits or further breakdown of the remaining data. |
| Hardware Specification | No | No specific hardware details (like GPU/CPU models, memory, or cloud instance types) used for running experiments were mentioned. |
| Software Dependencies | No | No specific software dependencies with version numbers (e.g., programming languages, libraries, or solvers) were mentioned. |
| Experiment Setup | Yes | The optimization problem in Equation 8 is solved to predict the difficulty of the images in X L, where the kernel function Φ is set to be the linear kernel, α = 10 4 and β = 3. The parameter η is set between 0.1 and 0.3. |