Improving Learning-from-Crowds through Expert Validation
Authors: Mengchen Liu, Liu Jiang, Junlin Liu, Xiting Wang, Jun Zhu, Shixia Liu
IJCAI 2017 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Experiments on both real-world and simulated datasets indicate that given a speciļ¬c accuracy goal (e.g., 95%), our method reduces expert effort from 39% to 60% compared with the state-of-the-art method. |
| Researcher Affiliation | Collaboration | Mengchen Liu1,4, Liu Jiang1,4, Junlin Liu1,4, Xiting Wang2, Jun Zhu3,4 and Shixia Liu1,4 1School of Software, Tsinghua University, Beijing, P.R. China 2Microsoft Research, Beijing, P.R. China 3Department of Computer Science and Technology, Tsinghua University, Beijing, P.R. China 4Tsinghua National Lab for Information Science and Technology |
| Pseudocode | No | No, the paper describes algorithmic steps in narrative text but does not include any clearly labeled pseudocode or algorithm blocks. |
| Open Source Code | No | No, the paper does not provide an explicit statement or link for the open-sourcing of its methodology code. |
| Open Datasets | Yes | We used the following datasets in our experiments. Dog [Zhou et al., 2012]: It contains 800 images of 4 breeds of dogs from Image Net [Deng et al., 2009]. Age [Han et al., 2015]: It contains 1,002 face images. Monkey: It contains images of 4 kinds of wild monkeys (Siamang, Guenon, Patas and Baboon). These images were selected from Image Net and we simulated the crowdsourced labels for each image by the method used in [Hung et al., 2013]. News: It contains documents of 4 topics from the 20News Group dataset [Lang, 1995]. |
| Dataset Splits | No | No, the paper does not specify exact percentages or sample counts for training, validation, or test dataset splits. |
| Hardware Specification | Yes | All the experiments were conducted on a workstation with Intel Core i5 CPU (3.3 GHz) and 16 GB of Memory. |
| Software Dependencies | No | No, the paper mentions software components like M3V model, VGG-NET, and TF-IDF but does not provide specific version numbers for any software dependencies. |
| Experiment Setup | Yes | In our implementation, p(Uc), p(Uf), p(Um), and p(Us) are set to 0.36, 0.04, 0.54, and 0.06, respectively. Here mi,l denotes the number of Gibbs samplers that return l as the estimated label for instance i and m (m = 5 in our implementation) is the number of Gibbs samplers used. For an image, we extracted its feature vector by using a deep convolutional neural network: VGG-NET [Simonyan and Zisserman, 2014]. We used the output of the last but one fully-connected layer as the feature vector of the image. For a document, we extracted its feature vector by using TF-IDF. |