Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in Coakley et alK. L. Coakley, T. Snelleman, H. Hoos, and O. E. Gundersen, "The embrace of open science: An analysis of a decade of AI research and 56 800 conference papers," Under Review, 2026..
Crowdsourced Clustering: Querying Edges vs Triangles
Authors: Ramya Korlakai Vinayak, Babak Hassibi
NeurIPS 2016 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | In addition to theoretical justification, through several simulations and experiments on two real data sets on Amazon Mechanical Turk, we empirically demonstrate that, for a fixed budget, triangle queries uniformly outperform edge queries. |
| Researcher Affiliation | Academia | Ramya Korlakai Vinayak Department of Electrical Engineering Caltech, Pasadena EMAIL Babak Hassibi Department of Electrical Engineering Caltech, Pasadena EMAIL |
| Pseudocode | No | The paper describes algorithms but does not provide structured pseudocode or algorithm blocks. |
| Open Source Code | No | The paper does not provide any statement or link indicating that the source code for the described methodology is publicly available. |
| Open Datasets | Yes | 1. Dogs3 dataset has images of the following 3 breeds of dogs from the Stanford Dogs Dataset [28]: Norfolk Terrier (172), Toy Poodle (150) and Bouvier des Flanders (151), giving a total of 473 dogs images. ... 2. Birds5 dataset has 5 bird species from CUB-200-2011 dataset [29]: Laysan Albatross (60), Least Tern (60), Artic Tern (58), Cardinal (57) and Green Jay (57). |
| Dataset Splits | No | The paper does not explicitly provide training/test/validation dataset splits (percentages, counts, or specific methodology) for their experiments. |
| Hardware Specification | No | The paper does not provide specific hardware details (e.g., CPU/GPU models, memory, or cloud instance types) used for running its experiments. |
| Software Dependencies | No | The paper mentions software components and algorithms (e.g., k-means, Spectral Clustering, Amazon Mechanical Turk) but does not provide specific version numbers for any software dependencies. |
| Experiment Setup | Yes | We run the improved convex program (4.1) by setting λ = 1/ n. ... For edge queries, each HIT (Human Intelligence Task) has 30 queries of random pairs... For triangle queries, each HIT has 20 queries, with each query having 3 random images. |