Minimizing Time-to-Rank: A Learning and Recommendation Approach
Authors: Haoming Li, Sujoy Sikdar, Rohit Vaish, Junming Wang, Lirong Xia, Chaonan Ye
IJCAI 2019 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | On the practical side, our experiments on Amazon Mechanical Turk provide two interesting insights about user behavior: First, that users ranking strategies closely resemble selection or insertion sort, and second, that the time taken for a drag-and-drop operation depends linearly on the number of positions moved. These insights directly motivate our theoretical model of the optimization problem. We show that computing an optimal recommendation is NP-hard, and provide exact and approximation algorithms for a variety of special cases of the problem. Experimental evaluation on MTurk shows that, compared to a random recommendation strategy, the proposed approach reduces the average time-to-rank by up to 50%. |
| Researcher Affiliation | Academia | 1Duke University 2Rensselaer Polytechnic Institute 3Stanford University |
| Pseudocode | No | The paper describes algorithms and theoretical models but does not include any pseudocode or algorithm blocks. |
| Open Source Code | No | The paper does not explicitly state that source code for the methodology is available or provide a link. |
| Open Datasets | No | The paper describes using randomly generated lists of numbers on Amazon Mechanical Turk, but does not provide a specific link, DOI, or formal citation for a publicly available dataset. |
| Dataset Splits | Yes | To verify this, we used linear regression with timeto-rank (or sorting time) as the target variable and measured the mean squared error (MSE) using 5-fold cross-validation for three different choices of independent variables |
| Hardware Specification | No | The paper does not specify any hardware details used for running the experiments. |
| Software Dependencies | No | The paper does not mention any specific software or library names with version numbers. |
| Experiment Setup | Yes | We perform two sets of experiments on Amazon Mechanical Turk (MTurk). The first set of experiments (Section 5.1) is aimed at identifying the sorting strategies of the users as well as a model of their drag-and-drop behavior. The observations from these experiments directly motivate the formulation of our theoretical model, which we have already presented in Section 4. The second set of experiments (Section 5.2) is aimed at evaluating the practical usefulness of our approach. In both sets of experiments, the crowdworkers were asked to sort in increasing order randomly generated lists of numbers between 0 and 100. Sections 5.1 and 5.2 provide details about the length of the lists and how they are generated. In each poll, we recorded the time taken by a crowdworker to move an alternative (via drag-and-drop operation) and the number of positions by which the alternative was moved. To ensure data quality, we removed those workers from the data who failed to successfully order the integers more than 80% of the time, or did not complete all the polls. We also removed the workers with high variance in their sorting time; in particular, those with coefficient of variation above the 80th percentile. |