Optimize What You Evaluate With: Search Result Diversification Based on Metric Optimization
Authors: Hai-Tao Yu10399-10407
AAAI 2022 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | The experimental results on benchmark collections show that the proposed method achieves significantly improved performance over the state-of-the-art results. |
| Researcher Affiliation | Academia | Information Intelligence Lab Faculty of Library, Information and Media Science, University of Tsukuba yuhaitao@slis.tsukuba.ac.jp |
| Pseudocode | No | The paper describes the proposed method and includes a diagram (Figure 1), but no formal pseudocode or algorithm blocks are provided. |
| Open Source Code | Yes | Detailed implementation: https://github.com/wildltr/ptranking |
| Open Datasets | Yes | Four standard test collections released in the diversity tasks of TREC Web Track from 2009 to 2012 are adopted for the experiments (50 queries per each year). ... Specifically, the Clue Web09 Category B collection consisting of 50 million English web documents is used as the base. The doc2vec model is trained on all documents and the number of vector dimensions is set as 100. The initial vector representations of queries and documents can be obtained given the trained doc2vec model. Please refer to (Xia et al. 2017; Feng et al. 2018) for more details. |
| Dataset Splits | Yes | We perform 5-fold cross validation experiments following the same subset split as (Feng et al. 2018). In each fold, three subsets are used as the training data, the remaining two subsets are used as the validation data and the testing data. |
| Hardware Specification | No | The paper does not provide any specific details about the hardware (e.g., GPU/CPU models, memory) used for running the experiments. |
| Software Dependencies | No | The paper mentions optimizers (Adagrad) and activation functions (ReLU, GELU) but does not provide specific version numbers for these or other software libraries (e.g., Python, PyTorch versions). |
| Experiment Setup | Yes | The other hyperparameters are chosen via a grid search: number of attention heads {2, 4, 6}, number of self-attention layers {2, 4, 6}, learning rate {0.001, 0.01}, activation functions {Re LU, GELU}, global variance σ {0.1, 1, 10} and number of Gaussian components V {1, 10}. |