Near-Optimally Teaching the Crowd to Classify
Authors: Adish Singla, Ilija Bogunovic, Gabor Bartok, Amin Karbasi, Andreas Krause
ICML 2014 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Our experiments on simulated workers as well as three real image annotation tasks on Amazon Mechanical Turk show the effectiveness of our teaching algorithm. |
| Researcher Affiliation | Academia | Adish Singla ADISH.SINGLA@INF.ETHZ.CH Ilija Bogunovic* ILIJA.BOGUNOVIC@EPFL.CH G abor Bart ok BARTOK@INF.ETHZ.CH Amin Karbasi AMIN.KARBASI@INF.ETHZ.CH Andreas Krause KRAUSEA@ETHZ.CH ETH Z urich, Z urich, Switzerland * School of Computer and Communication Sciences, EPFL, Lausanne, Switzerland |
| Pseudocode | Yes | Policy 1 Teaching Policy STRICT |
| Open Source Code | No | The paper does not contain an explicit statement about releasing the source code for their methodology, nor does it provide a direct link to their own code repository. It only references a third-party tool's code. |
| Open Datasets | Yes | As our second dataset, we used a collection of 200 real images of four species of butterflies and moths from publicly available images (Imagenet), and 'We used a collection of 150 real images belonging to three species of woodpeckers from a publicly available dataset (Wah et al., 2011)'. |
| Dataset Splits | No | The paper describes training (teaching) and test sets, for example, 'We used 160 of these images (40 per sub-species) as teaching set X and the remaining 40 (10 per sub-species) for testing.' but does not explicitly mention a validation set or provide details about validation splits. |
| Hardware Specification | No | The paper does not provide specific details about the hardware (e.g., GPU/CPU models, memory) used for running its experiments. |
| Software Dependencies | No | The paper mentions 'CUBAM, implementing the approach of Welinder et al.' but does not provide specific version numbers for any software dependencies required to replicate the experiments. |
| Experiment Setup | Yes | We chose α = 2 for our algorithm STRICT. The test phase was set to 10 examples for the VW and BM tasks, and 16 examples for the WP task. We simulated 100 learners with varying α parameters chosen randomly from the set {2, 3, 4} and different initial hypotheses of the learners, sampled from H. |