Learning to Represent Programs with Property Signatures
Authors: Augustus Odena, Charles Sutton
ICLR 2020 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We discuss several potential applications of property signatures and show experimentally that they can be used to improve over a baseline synthesizer so that it emits twice as many programs in less than one-tenth of the time. We perform experiments on a new test set of 185 functional programs of varying difficulty, designed to be the sort of algorithmic problems that one would ask on an undergraduate computer science examination. We find that the use of property signatures leads to a dramatic improvement in the performance of the synthesizer, allowing it to synthesize over twice as many programs in less than one-tenth of the time (Section 4). |
| Researcher Affiliation | Industry | Augustus Odena, Charles Sutton Google Research {augustusodena,charlessutton}@google.com |
| Pseudocode | Yes | Figure 2: The top-down synthesizer that we use as a baseline in this work. In a loop until a satisfying program is found or we run out of time, we pop the lowest-cost partial program from the queue of all partial programs, then we fill in the holes in all ways allowed by the type system, pushing each new partial program back onto the queue. If there are no holes to fill, the program is complete, and we check it against the spec. The cost of a partial program is the sum of the costs of its pool elements, plus a lower bound on the cost of filling each of its typed holes, plus the sum of the costs of a few special operations such as tuple construction and lambda abstraction. |
| Open Source Code | Yes | We release2 the programming language, runtime environment, distributed search infrastructure, machine learning models, and training data from our experiments so that they can be used for future research. 2Available at https://github.com/brain-research/searcho |
| Open Datasets | Yes | We release2 the programming language, runtime environment, distributed search infrastructure, machine learning models, and training data from our experiments so that they can be used for future research. 2Available at https://github.com/brain-research/searcho |
| Dataset Splits | No | The paper describes how training data is generated and how test sets are constructed. For the experiment on predicting property signatures of function compositions, it states, 'divided the data into a training set of 45,000 elements and a test set of 5,000 elements'. However, it does not explicitly mention or specify a separate validation set or how it was used for hyperparameter tuning or model selection in the main experiments. |
| Hardware Specification | No | The paper does not provide any specific hardware details such as GPU or CPU models, memory specifications, or types of computing clusters used for running the experiments. It only mentions 'a distributed synthesis server'. |
| Software Dependencies | No | The paper describes their custom programming language 'Searcho' and its runtime, and mentions it's based on Niklas Een's Evo programming language. However, it does not specify version numbers for any underlying general-purpose software dependencies (e.g., Python, TensorFlow/PyTorch, CUDA, specific compilers, or operating systems) required to reproduce the experiments. |
| Experiment Setup | No | The paper describes high-level aspects of the experimental setup, such as randomly sampling configurations with fewer pool elements and sending them to a distributed synthesis server, and how pool elements are sampled in proportion to model predictions. It also describes the neural network as 'fully connected, with learned embeddings'. However, it lacks specific numerical hyperparameters such as learning rates, batch sizes, optimizer types, number of training epochs, or detailed network architecture (number of layers, nodes per layer, activation functions) for the machine learning models used in the experiments. |