Deep Learning from Crowds
Authors: Filipe Rodrigues, Francisco Pereira
AAAI 2018 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We empirically show that the proposed approach is able to internally capture the reliability and biases of different annotators and achieve new state-of-the-art results for various crowdsourced datasets across different settings, namely classification, regression and sequence labeling. |
| Researcher Affiliation | Academia | Filipe Rodrigues, Francisco C. Pereira Dept. of Management Engineering, Technical University of Denmark Bygning 116B, 2800 Kgs. Lyngby, Denmark rodr@dtu.dk, camara@dtu.dk |
| Pseudocode | No | The paper describes algorithms in text, but no structured pseudocode or algorithm blocks were found. |
| Open Source Code | Yes | Source code, datasets and demos for all experiments are provided at: http://www.fprodrigues.com/. |
| Open Datasets | Yes | Source code, datasets and demos for all experiments are provided at: http://www.fprodrigues.com/. (Kaggle 2013); (Rodrigues et al. 2017); (Russell et al. 2008); (Rodrigues, Pereira, and Ribeiro 2013); (Sang and Meulder 2003) |
| Dataset Splits | Yes | The base architecture was selected from a set of possible configurations using the true labels by optimizing the accuracy on a validation set (consisting of 20% of the train set) through random search. |
| Hardware Specification | Yes | reducing the training time by at least one order of magnitude when compared to the latter (minutes instead of several hours on a Core i7 with 32GB of RAM and a NVIDIA GTX 1070). |
| Software Dependencies | No | The proposed crowd layer (CL) was implemented as a new type of layer in Keras (Chollet 2015), so that using it in practice requires only a single line of code. |
| Experiment Setup | Yes | For this particular problem we used a fairly standard CNN architecture with 4 convolutional layers with 3x3 patches, 2x2 max pooling and Re LU activations. The output of the convolutional layers is then fed to a fully-connected (FC) layer with 128 Re LU units and finally goes to an output layer with a softmax activation. We use batch normalization (Ioffe and Szegedy 2015) and apply 50% dropout between the FC and output layers. |