Active Learning for Top-$K$ Rank Aggregation from Noisy Comparisons
Authors: Soheil Mohajer, Changho Suh, Adel Elmahdy
ICML 2017 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | In this section, we empirically evaluate the performance of the proposed algorithm by conducting Monte Carlo simulations on synthetic data and developing a benchmark comparison against the state-of-the-art active and passive ranking algorithms in the literature. In an effort to guarantee fairness in the performance comparison, we jointly investigate the average number of pairwise comparisons and the corresponding empirical success rate of identifying top-K items. The source code of our algorithm4 is provided to allow for reproducible research. |
| Researcher Affiliation | Academia | 1ECE, University of Minnesota, Twin Cities, MN, USA. 2EE, KAIST, Daejeon, South Korea. |
| Pseudocode | Yes | Algorithm 1 SELECT(X; m) and Algorithm 2 TOP(X; K, m) |
| Open Source Code | Yes | The source code of our algorithm4 is provided to allow for reproducible research.4The source code is accessible via Git Hub at https://github.com/a-elmahdy/Active-Learning-from-Noisy-Comparisons.git |
| Open Datasets | No | The paper mentions using 'synthetic data' for Monte Carlo simulations but does not provide concrete access information (e.g., links, citations) for this data or its generation parameters. |
| Dataset Splits | No | The paper evaluates performance using Monte Carlo simulations on synthetic data but does not specify explicit training, validation, or test dataset splits in terms of percentages or sample counts. |
| Hardware Specification | No | The paper does not provide specific hardware details (e.g., GPU/CPU models, memory, cloud instances) used for running its experiments. |
| Software Dependencies | No | The paper mentions providing source code but does not list specific software dependencies with version numbers required to replicate the experiments. |
| Experiment Setup | Yes | The plots in Fig. 3(a) indicate the performance of different active ranking algorithms under the uniform noise model to identify the top-1 and top-16 items, respectively, when n = 210 = 1024, K = γ2 = 0.0225 and δ = 0.1 (Heckel algorithm parameter); (b) Active algorithms under BTL model for different values of K when n = 1024, K = 0.0225, wi ∈ (0, 3], 1 ≤ i ≤ n, and δ = 0.1 (Heckel algorithm parameter); (c) Performance Comparison between the proposed Top-K algorithm versus the Copeland Counting algorithm (with α = 8 and p = 0.6) under the uniform noise model for different values of K when n = 128, K = γ2 = 0.1. |