Obtaining Calibrated Probabilities with Personalized Ranking Models
Authors: Wonbin Kweon, SeongKu Kang, Hwanjo Yu4083-4091
AAAI 2022 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Extensive evaluations with various personalized ranking models on real-world datasets show that both the proposed calibration methods and the unbiased empirical risk minimization significantly improve the calibration performance. |
| Researcher Affiliation | Academia | Wonbin Kweon, Seong Ku Kang, Hwanjo Yu* Pohang University of Science and Technology, South Korea |
| Pseudocode | No | The paper does not contain any sections or figures explicitly labeled 'Pseudocode' or 'Algorithm'. |
| Open Source Code | Yes | Our source code is publicly available2. 2https://github.com/Wonbin Kweon/Calibrated Ranking Models AAAI2022 |
| Open Datasets | Yes | To the best of our knowledge, there are two real-world datasets that have separate unbiased test sets where the users are asked to rate uniformly sampled items (i.e., Ou,i = 1 for test sets). Yahoo!R33 has over 300K interactions in the training set and 54K preferences in the test set from 15.4K users and 1K songs. Coat (Schnabel et al. 2016) has over 7K interactions in the training set and 4.6K preferences in the test set from 290 users and 300 coats. |
| Dataset Splits | Yes | We hold out 10% of the training set as the validation set for the hyperparameter tuning of the base models and the optimization of the calibration methods. |
| Hardware Specification | No | The paper does not provide any specific details about the hardware used to run the experiments. |
| Software Dependencies | No | The paper mentions using 'the existing module of Scipy (Pedregosa et al. 2011)' but does not provide specific version numbers for Scipy or any other software dependencies. |
| Experiment Setup | No | The paper states 'The details for the training of these base models can be found in Appendix E.' indicating that specific experimental setup details are not present in the main text. |