Learning to Truncate Ranked Lists for Information Retrieval
Authors: Chen Wu, Ruqing Zhang, Jiafeng Guo, Yixing Fan, Yanyan Lan, Xueqi Cheng4453-4461
AAAI 2021 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Empirical results over the Robust04 and MQ2007 datasets demonstrate the effectiveness of our approach as compared with the state-of-the-art baselines. |
| Researcher Affiliation | Academia | CAS Key Lab of Network Data Science and Technology, Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China University of Chinese Academy of Sciences, Beijing, China {wuchen17z, zhangruqing, guojiafeng, fanyixing, lanyanyan, cxq}@ict.ac.cn |
| Pseudocode | No | The paper describes its model and training process using text and mathematical formulas, but it does not include pseudocode or an algorithm block. |
| Open Source Code | No | The paper states 'We implement our Attn Cut model in Py Torch2.' with a footnote linking to the PyTorch website, but there is no explicit statement or link indicating that their specific implementation code is open-source. |
| Open Datasets | Yes | We conduct experiments on two representative IR datasets. Robust04 contains 250 queries and 528k news articles, whose topics are collected from TREC 2004 Robust Track1. There are about 70 relevant documents (news articles) for each query. 1https://trec.nist.gov/data/robust.html ... Million Query Track 2007 (MQ2007) is a LETOR (Qin et al. 2010) benchmark dataset which uses Gov2 web collection. |
| Dataset Splits | No | For two datasets, we randomly divide them into a training set (80%) and a testing set (20%) following (Lien, Cohen, and Croft 2019) to achieve comparable performance. No explicit mention of a validation set. |
| Hardware Specification | No | The paper does not provide specific hardware details (e.g., GPU/CPU models, memory) used for running its experiments, only mentioning implementation in PyTorch. |
| Software Dependencies | No | The paper mentions using PyTorch and gensim, but it does not specify version numbers for these software dependencies, only providing general links to their websites. |
| Experiment Setup | Yes | For the Encoding Layer, we first compute the tf-idf and doc2vec of each document using gensim tool3 over the whole corpus. The dimension of tf-idf and doc2vec is 648730 and 200 respectively. The LSTM hidden unit size of the two-layer bi-directional LSTM is set as 128. For the Attention Layer, the hidden size t of the Transformer is 256 and the number h of selfattention heads is 4. For training, the mini-batch size for the update is set as 20 and 128 for Robust04 and MQ2007 respectively. The parameter τ for RAML learning is set as 0.95. We apply stochastic gradient decent method Adam (Kingma and Ba 2014) to learn the model parameters with the learning rate as 3 10 5. For recall-constraint model, we set the number of ordered bins B as 5. |