Efficient Online Crowdsourcing with Complex Annotations
Authors: Reshef Meir, Viet-An Nguyen, Xu Chen, Jagdish Ramakrishnan, Udi Weinsberg
AAAI 2024 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We conduct extensive evaluations on real-world crowdsourcing data from Meta and show the effectiveness of our proposed online algorithms in improving the cost-quality trade-off. and To showcase the effectiveness of our proposed methods, we conduct extensive numerical experiments on four complex annotation datasets and compare against several baselines. |
| Researcher Affiliation | Collaboration | Reshef Meir1,*, Viet-An Nguyen2, Xu Chen2, Jagdish Ramakrishnan2, Udi Weinsberg2 1Technion Israel Institute of Technology 2Central Applied Science, Meta |
| Pseudocode | Yes | Algorithm 1: OAK (LEARNING COMPONENT), Algorithm 2: OAK (ESTIMATION COMPONENT), Algorithm 3: POAK (ESTIMATION COMPONENT) |
| Open Source Code | No | The paper does not provide an explicit statement or a link indicating that the source code for the methodology is openly available. |
| Open Datasets | No | All methods are applied on four real crowdsourcing datasets obtained from Meta, covering a broad range of different labeling tasks. See basic information of the datasets in Tab. 3. The paper does not provide concrete access information (link, DOI, formal citation with authors/year) for these datasets. |
| Dataset Splits | No | We first randomly split each dataset into a training set and a test set. The paper explicitly mentions training and test sets, but not a separate validation set or the specific percentages/counts for the splits to reproduce the partitioning. |
| Hardware Specification | No | The paper does not provide any specific details about the hardware used for running the experiments. |
| Software Dependencies | No | The paper does not provide specific software dependency details with version numbers. |
| Experiment Setup | No | The paper mentions a meta-parameter 'γ = 10' and implies some aggregation function and thresholds are decided, but it does not provide specific details on hyperparameters, training configurations, or system-level settings for the experiments. |