Active Ranking without Strong Stochastic Transitivity
Authors: Hao Lou, Tao Jin, Yue Wu, Pan Xu, Quanquan Gu, Farzad Farnoud
NeurIPS 2022 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Thorough numerical experiments in various settings are conducted, demonstrating that Probe-Rank is significantly more sample-efficient than the state-of-the-art active ranking method. In this section, we present numerical experiments demonstrating the practical performance of Probe Rank. |
| Researcher Affiliation | Academia | Hao Lou Dept. of Electrical & Computer Engineering University of Virginia Charlottesville, VA 22903 haolou@virginia.edu Tao Jin Department of Computer Science University of Virginia Charlottesville, VA 22903 taoj@virginia.edu Yue Wu Department of Computer Science University of California, Los Angeles Los Angeles, CA 90095 ywu@cs.ucla.edu Pan Xu Dept. of Biostatistics & Bioinformatics Duke University Durham, NC 27705 pan.xu@duke.edu Quanquan Gu* Department of Computer Science University of California, Los Angeles Los Angeles, CA 90095 qgu@cs.ucla.edu Farzad Farnoud Dept. of Electrical & Computer Engineering University of Virginia Charlottesville, VA 22903 farzad@virginia.edu |
| Pseudocode | Yes | Subroutine 1 Successive-Comparison(i, j, δ, τ) (SC) / Algorithm 2 Probe-Rank / Subroutine 3 Probe-Max(S, δ) |
| Open Source Code | Yes | Our implementation can be found on Github 5. (Footnote 5: https://github.com/tao-j/aht/releases/tag/v0.1) |
| Open Datasets | No | The paper generates synthetic data based on various probabilistic comparison models and random ground truth rankings for its experiments. It does not use or provide concrete access to a pre-existing publicly available dataset. "The probabilistic comparison model pij is generated in different ways to satisfy different assumptions. Note that and d are tuning parameters in all the following settings." |
| Dataset Splits | No | The paper describes generating synthetic data for each trial and averaging results over 100 independent trials. It does not refer to traditional train/validation/test splits of a fixed dataset, as its focus is on active learning sample complexity rather than model training on a static dataset. |
| Hardware Specification | Yes | We use internal clusters of intel Skylake generation CPUs. |
| Software Dependencies | No | The paper mentions that the implementation is on GitHub but does not specify any particular software libraries, frameworks, or their version numbers (e.g., Python, PyTorch, TensorFlow versions). |
| Experiment Setup | Yes | The confidence level δ is fixed to be 0.1. / Specifically, we want to rank n items with the true ranking σ1 σ2 σn, where n varies over [10, 100]. / The probabilistic comparison model pij is generated in different ways to satisfy different assumptions. / The parameter d is set to be 0.3. / We choose α = 1 (see Figure 3(a)) and α = 1/2 (see Figure 3(b)). / In Figure 4, we fix n = 80 and let d vary. |