Disjunctive Program Synthesis: A Robust Approach to Programming by Example
Authors: Mohammad Raza, Sumit Gulwani
AAAI 2018 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We present an evaluation showing the significant increase in robustness achieved with our disjunctive approach, as illustrated by an increase from 59% to 93% of tasks for which correct programs can be learnt from a single example. |
| Researcher Affiliation | Industry | Mohammad Raza Microsoft Corporation One Microsoft Way Redmond, Washington, USA moraza@microsoft.com Sumit Gulwani Microsoft Corporation One Microsoft Way Redmond, Washington, USA sumitg@microsoft.com |
| Pseudocode | Yes | Figure 6: Disjunction operator semantics for symbol ψ, where T= ψ is the semantic type of ψ and Figure 7: Program synthesis algorithm (both figures contain pseudocode). |
| Open Source Code | No | We have implemented our DSLs and algorithm for disjunctive program synthesis for both the text and web extraction domains using the Microsoft PROSE framework for program synthesis1. 1https://microsoft.github.io/prose/ The paper mentions using an existing open-source framework for implementation but does not explicitly state that the code for the specific methodology described in this paper is open-sourced or provide a direct link to it. |
| Open Datasets | No | The benchmarks we obtained covered 47 extraction tasks from 8 different providers. The paper describes using benchmarks obtained from a product team, which implies an internal dataset. It does not provide any concrete access information (link, DOI, repository, or standard citation) for a publicly available or open dataset. |
| Dataset Splits | No | The measure of robustness we use is the minimum number of examples required to learn a correct a program for a given task. More precisely, the minimum number of examples is the minimum number of input-output instances that were required as examples to be given to the system such that the program it synthesized succeeded on all the remaining instances. While the paper refers to "examples" and "remaining instances" (effectively train and test), it does not explicitly provide specific dataset split information for training, validation, and testing (e.g., percentages, sample counts for each split, or references to predefined standard splits). |
| Hardware Specification | No | The paper does not provide any specific hardware details such as exact GPU/CPU models, processor types, or memory amounts used for running its experiments. |
| Software Dependencies | No | We have implemented our DSLs and algorithm for disjunctive program synthesis for both the text and web extraction domains using the Microsoft PROSE framework for program synthesis1. 1https://microsoft.github.io/prose/ The paper mentions using the Microsoft PROSE framework, but it does not provide specific version numbers for any ancillary software dependencies (e.g., programming languages, libraries, or other tools). |
| Experiment Setup | Yes | In our experiments we found a cut-off of top 10 disjuncts yields an effective balance of performance and robustness in practice. |