Transformers as Soft Reasoners over Language
Authors: Peter Clark, Oyvind Tafjord, Kyle Richardson
IJCAI 2020 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We train transformers to reason (or emulate reasoning) over these sentences using synthetically generated data. Our models, that we call Rule Takers, provide the first empirical demonstration that this kind of soft reasoning over language is learnable, can achieve high (99%) accuracy, and generalizes to test data requiring substantially deeper chaining than seen during training (95%+ scores). |
| Researcher Affiliation | Industry | Peter Clark , Oyvind Tafjord and Kyle Richardson Allen Institute for AI, Seattle, WA {peterc,oyvindt,kyler}@allenai.org |
| Pseudocode | No | The paper does not contain any structured pseudocode or algorithm blocks. |
| Open Source Code | Yes | A live demo and all our datasets are available at https://allenai.org/data/ruletaker |
| Open Datasets | Yes | Each dataset contains 100k examples (25k of each Type without/with negation)... A live demo and all our datasets are available at https://allenai.org/data/ruletaker |
| Dataset Splits | Yes | Data is randomly split 70/10/20 into train/dev/test partitions, ensuring no overlap of theories between each partition. |
| Hardware Specification | No | The paper does not provide specific hardware details (e.g., GPU/CPU models, memory, or cloud instance types) used for running the experiments. |
| Software Dependencies | No | The paper mentions models like RoBERTa and BERT, but does not specify software versions for libraries, frameworks, or programming languages (e.g., Python, PyTorch, TensorFlow versions). |
| Experiment Setup | No | We use fixed hyperparameters (learning rate etc), inheriting the settings from Ro BERTa on RACE [Liu et al., 2019]. |