Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in [1].
QORA: Zero-Shot Transfer via Interpretable Object-Relational Model Learning
Authors: Gabriel Stella, Dmitri Loguinov
ICML 2024 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | In our test domains, QORA achieves 100% predictive accuracy using almost four orders of magnitude fewer observations than a neural-network baseline, demonstrates zero-shot transfer to modified environments, and adapts rapidly when applied to tasks involving previously unseen object interactions. |
| Researcher Affiliation | Academia | Gabriel Stella 1 Dmitri Loguinov 1 1Department of Computer Science & Engineering, Texas A&M University, College Station, TX, USA. |
| Pseudocode | Yes | Thus, throughout this section, we maintain a high-level discussion; where relevant, we refer to pseudocode listings in Appendix A, which also contains more detailed information about the algorithm. ... A. QORA Pseudocode |
| Open Source Code | Yes | The source code of both QORA s reference implementation and our benchmark suite are available online (Stella, 2024). |
| Open Datasets | No | Our benchmark environments are described in Appendix B. ... Each domain has several parameters (e.g., width, height) that control the initial states it will generate. |
| Dataset Splits | No | The paper describes generating observations and training, but does not specify explicit train/validation/test dataset splits (e.g., percentages or counts) or cross-validation setup. |
| Hardware Specification | No | Finally, we compare QORA s runtime to that of a standard neural-network implementation, Py Torch, on the same machine. Note that for fairness, Py Torch is run on the CPU. |
| Software Dependencies | No | The MHDPA baseline consisted of a single 5-head dot-product attention layer (as implemented by Py Torch)... |
| Experiment Setup | Yes | For both NPE and MHDPA, we used 10 batches of 100 observations per epoch. The learning rate was 0.01 for NPE and 0.005 for MHDPA. We used stochastic gradient descent with momentum of 0.9 and L1 loss (since outputs were linear). |