Fact Checking in Community Forums
Authors: Tsvetomila Mihaylova, Preslav Nakov, Lluís Màrquez, Alberto Barrón-Cedeño, Mitra Mohtarami, Georgi Karadzhov, James Glass
AAAI 2018 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Evaluation results show a MAP value of 86.54, which is 21 points absolute above the baseline. and Evaluation and Results Settings We train an SVM classifier (Joachims 1999) on the 249 examples as described above, where each example is one question answer pair. For the evaluation, we use leave-one-thread-out cross validation, where each time we exclude and use for testing one of the 71 questions together with all its answers. We report Accuracy, Precision, Recall, and F1 for the classification setting. We also calculate Mean Average Precision (MAP). |
| Researcher Affiliation | Collaboration | Tsvetomila Mihaylova,1 Preslav Nakov,2 Llu ıs M arquez,2 Alberto Barr on-Cede no,2 Mitra Mohtarami,3 Georgi Karadzhov,1 James Glass 3 1Sofia University St. Kliment Ohridski , Sofia, Bulgaria 2Qatar Computing Research Institute, Hamad bin Khalifa University, Doha, Qatar 3Massachusetts Institute of Technology, Cambridge, MA, USA |
| Pseudocode | No | The paper does not contain any structured pseudocode or algorithm blocks. |
| Open Source Code | Yes | The dataset and the source code are available online at https: //github.com/qcri/QLFact Checking |
| Open Datasets | Yes | we create a new high-quality dataset CQA-QL-2016-fact , which we release to the research community;3 to the best of our knowledge, this is the first publicly-available dataset specifically targeting factuality in a c QA setting; and 3The dataset and the source code are available online at https: //github.com/qcri/QLFact Checking |
| Dataset Splits | Yes | Evaluation and Results Settings We train an SVM classifier (Joachims 1999) on the 249 examples as described above, where each example is one question answer pair. For the evaluation, we use leave-one-thread-out cross validation, where each time we exclude and use for testing one of the 71 questions together with all its answers. |
| Hardware Specification | No | The paper does not specify the hardware used for running the experiments. |
| Software Dependencies | No | The paper mentions using an 'SVM classifier (Joachims 1999)' but does not provide specific version numbers for any software dependencies or libraries. |
| Experiment Setup | No | The paper mentions training an SVM classifier but does not provide specific experimental setup details such as hyperparameters, optimizer settings, or other system-level training configurations. |