HodgeRank With Information Maximization for Crowdsourced Pairwise Ranking Aggregation
Authors: Qianqian Xu, Jiechao Xiong, Xi Chen, Qingming Huang, Yuan Yao
AAAI 2018 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Experiments show that the proposed methods boost the sampling efficiency as compared to traditional sampling schemes and are thus valuable to practical crowdsourcing experiments. |
| Researcher Affiliation | Collaboration | Qianqian Xu,1,3 Jiechao Xiong,2,3 Xi Chen,4 Qingming Huang,5,6 Yuan Yao7,3, 1 SKLOIS, Institute of Information Engineering, CAS, Beijing, China, 2 Tencent AI Lab, Shenzhen, China 3 BICMR and School of Mathematical Sciences, Peking University, Beijing, China 4 Department of IOMS, Stern School of Business, New York University, USA 5 University of Chinese Academy of Sciences, Beijing, China, 6 IIP., ICT., CAS, Beijing, China 7, Department of Mathematics, Hong Kong University of Science and Technology, Hong Kong |
| Pseudocode | Yes | Algorithm 1: Unsupervised active sampling algorithm. Input: An initial graph Laplacian L0 defined on the graph of n nodes. and Algorithm 2: Online supervised active sampling algorithm for binary comparison data. Input: Prior distribution parameters γ, μ0, L 1 0,γ. |
| Open Source Code | No | The paper does not contain a clear statement about open-sourcing the code for the described methodology or a link to a code repository. |
| Open Datasets | Yes | The first example gives a comparison of these three sampling schemes on a complete & balanced video quality assessment (VQA) dataset (Xu et al. 2011). It contains 38,400 paired comparisons of the LIVE dataset (LIV 2008) from 209 random observers. |
| Dataset Splits | No | The paper describes generating simulated data and using existing real-world datasets, but it does not specify explicit train/validation/test dataset splits (e.g., 80/10/10 percentages or sample counts). |
| Hardware Specification | Yes | All computation is done using MATLAB R2014a, on a Mac Pro desktop PC, with 2.8 GHz Intel Core i7-4558u, and 16 GB memory. |
| Software Dependencies | Yes | All computation is done using MATLAB R2014a |
| Experiment Setup | Yes | A crucial question here is how to choose γ in supervised active sampling experiments. In practice, for dense graph, we find that γ makes little difference in the experimental results, a smaller γ, say 0.01, or even 1e 5 is sufficient. However, for sparse graph such as reading level dataset, a bigger γ (i.e., γ = 1) may produce better performance. |