Write, Execute, Assess: Program Synthesis with a REPL
Authors: Kevin Ellis, Maxwell Nye, Yewen Pu, Felix Sosa, Josh Tenenbaum, Armando Solar-Lezama
NeurIPS 2019 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We validate our framework on two different domains (see Figure 1): inferring 2D and 3D graphics programs (in the style of computer aided design, or CAD) and synthesizing text-editing programs (in the style of Flash Fill [14]). In both cases we show that a code-writing agent equipped with a REPL and a value function to guide the search achieves faster, more reliable program synthesis. |
| Researcher Affiliation | Academia | Kevin Ellis MIT Maxwell Nye MIT Yewen Pu MIT Felix Sosa Harvard University Joshua B. Tenenbaum MIT Armando Solar-Lezama MIT |
| Pseudocode | No | The paper describes algorithms conceptually but does not contain structured pseudocode or algorithm blocks (e.g., labeled Algorithm X or Pseudocode). |
| Open Source Code | No | The paper does not provide an explicit statement or link to the open-source code for the methodology described in this paper. It mentions "Tao Du assisted by providing 3D ray tracing code, which we used when rerendering 3D programs" but this is not the authors' core method code. |
| Open Datasets | No | The paper mentions training on "randomly generated scenes" and using datasets from prior work ([6], [4], [19]), but it does not provide concrete access information (specific link, DOI, repository name, or formal citation with authors/year in brackets/parentheses for a publicly available or open dataset) for the data used for training. |
| Dataset Splits | No | The paper mentions training and testing, but it does not provide specific dataset split information (exact percentages, sample counts, or explicit mention of a validation split) needed to reproduce the data partitioning. |
| Hardware Specification | No | The paper mentions that "the sampling/resampling steps are easily batched on a GPU" but does not provide any specific hardware details such as GPU model, CPU type, memory, or cloud instance types, needed to reproduce the experiments. |
| Software Dependencies | No | The paper does not provide specific ancillary software details, such as library or solver names with version numbers, needed to replicate the experiment. |
| Experiment Setup | No | The paper describes the overall training strategy (e.g., pretraining, REINFORCE) but does not provide specific experimental setup details such as concrete hyperparameter values (e.g., learning rate, batch size, number of epochs) or precise training configurations in the main text. |