Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in [1].
On Inductive Abilities of Latent Factor Models for Relational Learning
Authors: Théo Trouillon, Eric Gaussier, Christopher R. Dance, Guillaume Bouchard
JAIR 2019 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We conduct an experimental survey of state-of-the-art models, not towards a purely comparative end, but as a means to get insight about their inductive abilities. To assess the strengths and weaknesses of each model, we create simple tasks that exhibit first, atomic properties of binary relations, and then, common inter-relational inference through synthetic genealogies. Based on these experimental results, we propose new research directions to improve on existing models. |
| Researcher Affiliation | Collaboration | Th eo Trouillon EMAIL Univ. Grenoble Alpes 700 avenue Centrale 38401 Saint Martin d H eres, France Eric Gaussier EMAIL Univ. Grenoble Alpes 700 avenue Centrale 38401 Saint Martin d H eres, France Christopher R. Dance EMAIL NAVER LABS Europe 6 chemin de Maupertuis 38240 Meylan, France Guillaume Bouchard EMAIL Facebook 1 Rathbone Square WT1 1FB London, United Kingdom |
| Pseudocode | Yes | Appendix A. Learning Algorithm Algorithm 1 describes the stochastic gradient descent algorithm used to learn the evaluated models, with the Ada Grad learning-rate updates (Duchi et al., 2011). ... Algorithm 1 Stochastic gradient descent with Ada Grad |
| Open Source Code | No | Section 3.3 states: "All data sets are made available1. 1 https://github.com/ttrouill/induction_experiments". This link specifically refers to the datasets, not the source code for the methodology described in the paper. |
| Open Datasets | Yes | All data sets are made available1. 1 https://github.com/ttrouill/induction_experiments |
| Dataset Splits | Yes | We conduct here a 10-fold cross-validation (CV) with 80% training, 10% validation and 10% test... We propose to split the data in three different ways to explore inductive abilities of the models. The first split is the classical random split... In the second split... In this split... (Figure 9 shows tensor drawings of the three splits.) For each split we explore different values of p {0.8, 0.4, 0.2, 0.1}. We also run with p = 0 in the last (family) split... The number of facts in the training, validation and test sets of each split are summarized in Table 5. |
| Hardware Specification | No | The paper does not provide specific hardware details (e.g., CPU, GPU models, memory, or processor types) used for running the experiments. It focuses on the models, experimental design, and results. |
| Software Dependencies | No | The paper mentions optimization algorithms like "stochastic gradient descent with mini-batches" and "Ada Grad (Duchi, Hazan, & Singer, 2011)", but it does not specify any software libraries, frameworks, or their version numbers (e.g., Python, PyTorch, TensorFlow, scikit-learn versions). |
| Experiment Setup | Yes | We minimized the negative log-likelihood of the logistic model with L2 regularization applied entity-wise and relation-wise over their vector embeddings... The loss is optimized through stochastic gradient descent with mini-batches (10 batches for the relation properties experiment, and 100 for the families experiment), Ada Grad (Duchi, Hazan, & Singer, 2011) with an initial learning rate of 0.1, and early stopping when average precision decreased on the validation set calculated every 50 epochs. The λ regularization parameter was validated over the values {0.3, 0.1, 0.03, 0.01, 0.003, 0.001, 0.0003, 0.0} for each model for each factorization rank K. Parameters are initialized from a centered unit-variance Gaussian distribution. The complete algorithm is detailed in Appendix A. ... For the Trans E model... we enforced entity embeddings to have unit norm ||ei||2 = 1 for all i E... With the F model, prediction of unobserved entity pairs in the training set is handled through random Gaussian embeddings. |