Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in Coakley et alK. L. Coakley, T. Snelleman, H. Hoos, and O. E. Gundersen, "The embrace of open science: An analysis of a decade of AI research and 56 800 conference papers," Under Review, 2026..
Learning Task Specifications from Demonstrations
Authors: Marcell Vazquez-Chanlatte, Susmit Jha, Ashish Tiwari, Mark K. Ho, Sanjit Seshia
NeurIPS 2018 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | In our experiments, we demonstrate how learning specifications can help avoid common problems that often arise due to ad-hoc reward composition.Running a fairly unoptimized implementation of Algorithm 2 on the concept class and demonstrations took approximately 95 seconds and resulted in 172 eϕ queries ( 18% of the concept class). |
| Researcher Affiliation | Collaboration | 1 University of California, Berkeley 2 SRI International, Menlo Park |
| Pseudocode | Yes | Algorithm 1 Inference on chains and Algorithm 2 Inference on Partial Orders |
| Open Source Code | No | No explicit statement or link providing access to the source code for the methodology described in this paper was found. |
| Open Datasets | No | The paper uses custom demonstrations shown in Figure 1 for its experiments but does not provide concrete access information (link, DOI, or formal citation) for a publicly available or open dataset. |
| Dataset Splits | No | The paper uses a set of demonstrations for inference but does not specify any explicit training, validation, or test dataset splits. |
| Hardware Specification | No | No specific hardware details (e.g., GPU/CPU models, memory) used for running experiments are provided in the paper. |
| Software Dependencies | No | The paper mentions using a "SAT solver" and "Binary Decision Diagram (BDD)" but does not specify version numbers for these or any other software dependencies. |
| Experiment Setup | No | The paper describes the scenario and concept class but does not provide specific experimental setup details such as hyperparameters, training configurations, or other system-level settings. |