Grounding Spatio-Temporal Language with Transformers
Authors: Tristan Karch, Laetitia Teodorescu, Katja Hofmann, Clément Moulin-Frier, Pierre-Yves Oudeyer
NeurIPS 2021 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We test models on two classes of generalization: 1) generalization to randomly held-out sentences; 2) generalization to grammar primitives. We observe that maintaining object identity in the attention computation of our Transformers is instrumental to achieving good performance on generalization overall, and that summarizing object traces in a single token has little influence on performance. |
| Researcher Affiliation | Collaboration | Inria Flowers Team Université de Bordeaux firstname.lastname@inria.fr Katja Hofmann Microsoft Research Cambridge, UK |
| Pseudocode | No | The paper describes the architectures and operations but does not include any clearly labeled pseudocode or algorithm blocks. |
| Open Source Code | Yes | We also release our code under open-source license as well as pretrained models and datasets to encourage the wider community to build upon and extend our work in the future. Links. The source code as well as the generated datasets can be found at https://github.com/flowersteam/spatio-temporal-language-transformers |
| Open Datasets | Yes | In the absence of any dedicated dataset providing spatio-temporal descriptions from behavioral traces of an agent, we introduce Temporal Playground (Fig. 1) an environment coupled with a templated grammar designed to study spatio-temporal language grounding. ... Links. The source code as well as the generated datasets can be found at https://github.com/flowersteam/spatio-temporal-language-transformers |
| Dataset Splits | No | The paper specifies train and test set usage ("We remove 15% of all possible sentences in each category from the train set and evaluate the F1 score on those sentences."), and notes that the "testing set" was used for hyperparameter search ("We extract the best condition for each model by measuring the mean F1 on a testing set..."), but does not explicitly mention a separate "validation" split. |
| Hardware Specification | No | The paper states, "This work was performed using HPC resources from GENCI-IDRIS (Grant 2020-A0091011996)", but does not provide specific hardware details such as GPU/CPU models, processor types, or memory. |
| Software Dependencies | No | The paper mentions various models and optimizers (e.g., Transformers, LSTMs, Adam) but does not provide specific version numbers for any software dependencies or libraries used. |
| Experiment Setup | No | The paper states, "Details about the training procedure and the hyper parameter search are provided in Supplementary Section B.4.", but does not explicitly provide specific hyperparameter values or detailed training configurations within the main body of the paper. |