Multi-Task Learning for Document Ranking and Query Suggestion
Authors: Wasi Uddin Ahmad, Kai-Wei Chang, Hongning Wang
ICLR 2018 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We evaluate the effectiveness of the proposed framework using the publicly available AOL search log and compare with several well-known classical retrieval models, as well as various neural retrieval models specifically designed for ad-hoc retrieval. We conduct our experiments on the publicly available AOL search log (Pass et al., 2006). In the document ranking task, we need to rank the most relevant (e.g., most clickable) document on top. We used three standard ranking metrics, mean average precision (MAP), mean reciprocal rank (MRR) and normalized discounted cumulative gain (NDCG) metric computed at positions one, three, five and ten, to measure the performance. |
| Researcher Affiliation | Academia | Wasi Uddin Ahmad & Kai-Wei Chang Department of Computer Science University of California, Los Angeles {wasiahmad,kwchang}@cs.ucla.edu Hongning Wang Department of Computer Science University of Virginia hw5x@virginia.edu |
| Pseudocode | No | The paper provides architectural diagrams (Figures 1, 3, 4) but does not include any structured pseudocode or algorithm blocks. |
| Open Source Code | Yes | 2. We provide detailed experiment analysis, and release the implementation1 and the data processing tool to facilitate future research. 1https://github.com/wasiahmad/mnsrf |
| Open Datasets | Yes | We conduct our experiments on the publicly available AOL search log (Pass et al., 2006). |
| Dataset Splits | Yes | We randomly selected 1,032,459 sessions for training, 129,053 sessions for development and 91,108 sessions for testing, with no overlapping. In total, there are 2,987,486 queries for training, 287,138 for development, and 259,117 for testing. |
| Hardware Specification | Yes | M-NSRF is implemented in Py Torch and it runs on a single GPU (TITAN X) with roughly a runtime of 90 minutes per epoch. |
| Software Dependencies | No | The paper mentions 'implemented in Py Torch' and uses 'Glo Ve word embeddings', but does not provide specific version numbers for these software components. |
| Experiment Setup | Yes | The model was trained end-to-end and we used mini-batch SGD with Adam (Kingma & Ba, 2014) for optimization. with the two momentum parameters set to 0.9 and 0.999 respectively. We use 300-dimensional word vectors... All training used a mini-batch size of 32 to fit in single GPU memory. Learning rate was fixed to 0.001. We used dropout (0.20) (Srivastava et al., 2014) and early stopping with a patience of 5 epochs for regularization. We set λ = 0.1 for entropy regularization in Eq. (3). |