Variational Inference for Learning Representations of Natural Language Edits
Authors: Edison Marrese-Taylor, Machel Reid, Yutaka Matsuo13552-13560
AAAI 2021 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Our results indicate that evaluation metrics that are related to the task used to obtain edit representations are generally good predictors for the performance of these representations in downstream tasks, although not always. Compared to existing approaches, our model obtains better scores on the intrinsic evaluation, and the representations obtained by our approach can also consistently deliver better performance in our set of introduced downstream tasks. Our code and data are available on Git Hub1. |
| Researcher Affiliation | Academia | Edison Marrese-Taylor, Machel Reid, Yutaka Matsuo Graduate School of Engineering, The University of Tokyo {emarrese,machelreid,matsuo}@weblab.t.u-tokyo.ac.jp |
| Pseudocode | No | The paper describes the generative process with mathematical equations and defines neural network components, but it does not include any clearly labeled pseudocode blocks or algorithms. |
| Open Source Code | Yes | Our code and data are available on Git Hub1. 1https://github.com/epochx/PEER |
| Open Datasets | Yes | Table 1 provides a descriptive summary of the datasets included in PEER, and we provide details about each below: Wiki Atomic Sample... (Faruqui et al. 2018). QT21 De-En... (Specia et al. 2017). Lang-8 Corpus of Learner English... (Mizumoto et al. 2012). W&I + LOCNESS... (Bryant et al. 2019). |
| Dataset Splits | Yes | We evaluated each model variation in both the intrinsic and extrinsic tasks using the validation set on each case. Table 2 summarizes the results of our ablation experiments. Results show the effectiveness of the introduced x loss, which consistently helps the baseline model obtain better performance. The fact that performance not only improves on the intrinsic tasks, but also on the extrinsic evaluation, suggests that this technique effectively helps the latent code store meaningful information about the edit. |
| Hardware Specification | No | The paper mentions: 'Acknowledgements We are grateful to the NVIDIA Corporation, which donated two of the GPUs used for this research.' While it mentions the company and the type of hardware (GPUs), it does not specify exact models (e.g., NVIDIA Tesla V100, RTX 3090) or other detailed specifications like CPU model, memory, or specific cloud instance types. |
| Software Dependencies | No | The paper describes various model architectures like LSTMs, VAEs, and discusses concepts like word dropout and KL annealing. It also mentions adapting 'code release' from another paper. However, it does not specify particular software versions (e.g., 'Python 3.x', 'PyTorch 1.x', or specific versions of other libraries/frameworks) that would be needed for replication. |
| Experiment Setup | Yes | In principle, we follow previous work (Kingma and Welling 2014) and set the number of samples to 1 given that we train our model with a minibatch size that is large enough. To deal with these issues, we utilize word dropout, and we anneal the KL term in the loss utilizing a sigmoid function, following the work of Bowman et al. (2016). Additionally, we also follow recent work of Li et al. (2019), who discovered that when the inference network of a text VAE is initialized with the parameters of an encoder that is pre-trained using an auto-encoder objective, the VAE model does not suffer from the posterior collapse problem. Therefore, our model is first trained with zero KL weight until convergence. Then, the decoder is reset, and the whole model is re-trained. |