Which Tricks are Important for Learning to Rank?
Authors: Ivan Lyzhin, Aleksei Ustimenko, Andrey Gulin, Liudmila Prokhorenkova
ICML 2023 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We provide a theoretical explanation of their differences and extensive empirical evaluation. In Section 4, we conduct a thorough comparison of existing LTR algorithms on several benchmarks, show that Yeti Loss outperforms the competitors for specific ranking quality functions, and analyze the effect of the main algorithmic details on the quality of LTR. |
| Researcher Affiliation | Industry | Ivan Lyzhin 1 Aleksei Ustimenko 2 Andrey Gulin 1 Liudmila Prokhorenkova 3 1Yandex, Moscow, Russia 2Share Chat, London, UK 3Yandex Research, Amsterdam, The Netherlands. |
| Pseudocode | No | The paper describes algorithms and methods but does not include any explicitly labeled pseudocode or algorithm blocks. |
| Open Source Code | No | The paper mentions using and modifying existing open-source libraries (Cat Boost, Light GBM) but does not state that the authors' own code for their methodology (Yeti Loss modifications or experiment scripts) is open-source or provide a link. |
| Open Datasets | Yes | Datasets We use six publicly available datasets. The first two are Web10K and Web30K released by Microsoft (Qin & Liu, 2013). Following previous studies (Qin et al., 2021; Ustimenko & Prokhorenkova, 2020; Wang et al., 2018), we use Fold 1 for these two datasets. We also use two datasets from YAHOO! Learning to Rank Challenge (Chapelle & Chang, 2011). Finally, we take Istella and Istella-S datasets (Dato et al., 2016). |
| Dataset Splits | Yes | All datasets except for Istella are pre-divided into the train, validation, and test sets. For Istella, there is no standard validation set, so we randomly divided the train part into train and validation. Table 1 overviews the datasets used in the current study. |
| Hardware Specification | No | The paper does not provide specific details about the hardware (e.g., GPU models, CPU types) used to run the experiments. |
| Software Dependencies | No | The paper mentions using 'Cat Boost gradient boosting library' and 'Light GBM library' but does not specify exact version numbers for these or any other ancillary software components, which are required for full reproducibility. |
| Experiment Setup | Yes | For all algorithms, we set the maximum number of trees to 1000. We choose the best parameters, including the optimal number of trees, using the value of the desired loss function on the validation set. The list of tuned parameters is given in Appendix A. |