Scenario-based Question Answering with Interacting Contextual Properties
Authors: Haitian Sun, William W. Cohen, Ruslan Salakhutdinov
ICLR 2023 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We experiment with T-Reasoner on a synthetic dataset, Cond NLI, and two benchmark QA datasets, Conditional QA (Sun et al., 2021a) and Sh ARC (Saeidi et al., 2018), for scenario-based QA task. ... The experiment results are shown in Table 2. |
| Researcher Affiliation | Collaboration | Haitian Sun School of Computer Science Carnegie Mellon University haitians@cs.cmu.edu William W. Cohen Google Brain wcohen@google.com Ruslan Salakhutdinov School of Computer Science Carnegie Mellon University rsalakhu@cs.cmu.edu |
| Pseudocode | No | No pseudocode or algorithm blocks are present in the paper. The methodology is described in prose. |
| Open Source Code | Yes | Codes and data are available at https://github.com/haitian-sun/T-Reasoner. |
| Open Datasets | Yes | Codes and data are available at https://github.com/haitian-sun/T-Reasoner. We experiment with T-Reasoner on a synthetic dataset, Cond NLI, and two benchmark QA datasets, Conditional QA (Sun et al., 2021a) and Sh ARC (Saeidi et al., 2018), for scenario-based QA task. |
| Dataset Splits | Yes | Dataset statistics are shown in Table 11. Train Dev Test Sh ARC 15581 1622 5866 Conditional QA 2338 285 804 |
| Hardware Specification | No | The paper does not specify the hardware used for running experiments, such as specific GPU models, CPU models, or cloud computing instance types. |
| Software Dependencies | No | The paper mentions using pretrained LM checkpoints like T5, BART, and ELECTRA, but it does not specify version numbers for these software components or any underlying programming languages or libraries (e.g., Python, PyTorch/TensorFlow versions). |
| Experiment Setup | Yes | The paper states: "The number of Transformer layers in the reasoning module is a hyper-parameter. We choose the number of layers l = 3 or l = 4." It also mentions "The generation task is trained with teacher forcing." and "The final loss function is the sum of the answer loss lreason and the condition entailment loss lcond." These are specific setup details. |