Topic Concentration in Query Focused Summarization Datasets
Authors: Tal Baumel, Raphael Cohen, Michael Elhadad
AAAI 2016 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We define a task-based method to quantify topic concentration in datasets, i.e., the ratio of sentences within the dataset that are relevant to the query, and observe that the DUC 2005, 2006 and 2007 datasets suffer from very high topic concentration. We introduce TD-QFS, a new QFS dataset with controlled levels of topic concentration. We compare competitive baseline algorithms on TD-QFS and report strong improvement in ROUGE performance for algorithms that properly model query relevance as opposed to generic summarizers. |
| Researcher Affiliation | Academia | Tal Baumel, Raphael Cohen, Michael Elhadad Ben Gurion University, Dept. of Computer Science {talbau, cohenrap, elhadad}@cs.bgu.ac.il |
| Pseudocode | No | No, the paper does not contain structured pseudocode or algorithm blocks. |
| Open Source Code | No | No, the paper does not provide concrete access to source code for the methodology described. |
| Open Datasets | Yes | We introduce and make available a new dataset that we call the Topically Diverse QFS (TD-QFS) dataset to try to create a QFS benchmark with less topic concentration. The TD-QFS re-uses queries and document-sets from the Query Chain Focused Summarization (QCFS) (Baumel et al., 2014)... TD-QFS is freely available at http://www.cs.bgu.ac.il/~talbau/TDQFS/dataset.html |
| Dataset Splits | No | No, the paper does not provide specific train/validation/test dataset splits. It discusses processing portions of the document cluster for input to the summarizer, but not formal dataset splits for model training and evaluation. |
| Hardware Specification | No | No, the paper does not provide any specific hardware details used for running experiments. |
| Software Dependencies | No | No, the paper does not provide specific ancillary software details with version numbers. |
| Experiment Setup | Yes | We model QFS as a 2-stage process as illustrated in Fig.2: (1) rank passages in the cluster by similarity to the query; (2) filter the document cluster and apply a generic summarization algorithm on the most relevant passages. In our experiments, we use the KLSum method as the constant summarization method (Haghighi and Vanderwende, 2009)... For each retrieval model, we keep only the top-N sentences before applying the generic KLSum method so that we obtain variants with the top most-relevant passages containing up to 750, 1,000 2,250 words... output summarization length is 250 words. |