DetIE: Multilingual Open Information Extraction Inspired by Object Detection
Authors: Michael Vasilkovsky, Anton Alekseev, Valentin Malykh, Ilya Shenbin, Elena Tutubalina, Dmitriy Salikhov, Mikhail Stepnov, Andrey Chertok, Sergey Nikolenko11412-11420
AAAI 2022 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We present a novel single-pass method for Open IE inspired by object detection algorithms from computer vision. We use an order-agnostic loss based on bipartite matching that forces unique predictions and a Transformerbased encoder-only architecture for sequence labeling. The proposed approach is faster and shows superior or similar performance in comparison with state of the art models on standard benchmarks in terms of both quality metrics and inference time. Our model sets the new state of the art performance of 67.7% F1 on Ca RB evaluated as OIE2016 while being 3.35x faster at inference than previous state of the art. Experimental Setup We implement our model in pytorch lightning (Falcon 2019) with Hydra configuration framework (Yadan 2019). |
| Researcher Affiliation | Collaboration | 1 Skolkovo Institute of Science and Technology, Moscow, Russia 2 Huawei Noah s Ark lab, Moscow, Russia 3 St. Petersburg Department of Steklov Mathematical Institute of Russian Academy of Sciences, St. Petersburg, Russia 4 St. Petersburg State University, St. Petersburg, Russia 5 HSE University, Moscow, Russia 6 Kazan Federal University, Kazan, Russia 7 Sber AI, Moscow, Russia 8 Artificial Intelligence Research Institute, Moscow, Russia 9 ISP RAS Research Center for Trusted Artificial Intelligence, Moscow, Russia 10 Neuromation OU, Tallinn, Estonia |
| Pseudocode | No | The paper describes the model architecture and method conceptually and with diagrams (Figure 1, Figure 2), but does not provide pseudocode or an algorithm block. |
| Open Source Code | Yes | Code and models are available at https://github.com/sberbank-ai/Det IE. |
| Open Datasets | Yes | First, we use the recent LSOIE (Large-Scale Open Information Extraction) dataset (Solawetz and Larson 2021). For a fair comparison, we have also trained models on the dataset used by Kolluru et al. (2020a,b), called IMo JIE below. Another reason for training on two different datasets is that LSOIE differs in its annotation scheme from popular evaluation datasets such as OIE2016 (Stanovsky and Dagan 2016) and Ca RB (Bhardwaj, Aggarwal, and Mausam 2019); e.g., in LSOIE auxiliary verbs such as is or was are (intentionally) not included into predicates, while Ca RB adds more context into predicates than other OIE systems. We also experimented with adding Wikidata-based synthetic sentences during training (see below). |
| Dataset Splits | Yes | Dataset statistics are summarized in Table 1. In case of IMo JIE data, 10% of the samples are selected as validation. In case of LSOIE, the validation was performed on the test split of the dataset. We have analyzed the outputs of Det IEIMo JIE on a random sample of 100 sentences from the Ca RB validation set (Table 6). |
| Hardware Specification | Yes | Typical training time until the best model is reached is about 1.5 hours on an NVIDIA Tesla V100 GPU. batches of 32 processed on a single NVIDIA Tesla V100 GPU and 6 cores Intel(R) Xeon(R) CPU E5-2690 v4 @ 2.60GHz without any additional model optimizations. |
| Software Dependencies | No | The paper mentions software like 'pytorch lightning', 'Hydra', 'Hugging Face BERT', and 'Stanza', but does not specify their version numbers, which are necessary for full reproducibility. |
| Experiment Setup | Yes | Our best model was trained with Adam optimizer with learning rate 5e-4 and weight decay 1e-6, batch size 32, unfreezing 4 top layers of BERT, N = 20 detections, matching based on Io U similarity metrics, and doubled weights of non-background classes. |