Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in [1].
Terra: Imperative-Symbolic Co-Execution of Imperative Deep Learning Programs
Authors: Taebum Kim, Eunji Jeong, Geon-Woo Kim, Yunmo Koo, Sehoon Kim, Gyeongin Yu, Byung-Gon Chun
NeurIPS 2021 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We evaluated Terra s performance improvement and coverage with ten imperative DL programs for several DNN architectures. The results show that Terra can speed up the execution of all ten imperative DL programs, whereas Auto Graph, one of the state-of-the-art systems, fails to execute five of them. |
| Researcher Affiliation | Collaboration | Taebum Kim Seoul National University, Friendli AI EMAIL, EMAIL Eunji Jeong Samsung Research EMAIL Geon-Woo Kim Seoul National University, Friendli AI EMAIL, EMAIL Yunmo Koo Seoul National University, Friendli AI EMAIL, EMAIL Sehoon Kim University of California, Berkeley EMAIL Gyeong-In Yu Seoul National University EMAIL Byung-Gon Chun Seoul National University, Friendli AI EMAIL, EMAIL |
| Pseudocode | No | No explicitly labeled 'Pseudocode' or 'Algorithm' block was found in the provided text. Figure 1 shows code examples, but these are illustrative problem cases, not pseudocode for the proposed system. |
| Open Source Code | No | No explicit statement or link providing access to the open-source code for the methodology described in the paper was found. |
| Open Datasets | Yes | For the experiments, we use ten imperative DL programs collected from open-source Git Hub repositories: Drop Block [12], BERT-Q&A [13], Music Transformer [21], SDPoint [22], BERT-CLS [24], GPT2 [34], DCGAN [37], Res Net50 [38], Faster RCNN [44], and YOLOv3 [45]. |
| Dataset Splits | No | Experiment settings such as batch size and the dataset are included in Appendix E.' (Appendix E is not provided). No specific train/validation/test splits (e.g., 80/10/10) or absolute counts were mentioned for the datasets used. |
| Hardware Specification | Yes | We conduct all the experiments on a single machine that is equipped with 8-core AMD Ryzen 7 2700X @ 3.7GHz and an NVIDIA TITAN Xp GPU. |
| Software Dependencies | Yes | We use Tensor Flow [6] v2.4.1 as our baseline DL framework. We have built Terra on Tensor Flow v2.4.1... We use Ubuntu 18.04, CUDA 11.0, cu DNN 8.0, and Python 3.8.8. |
| Experiment Setup | No | The paper states 'Experiment settings such as batch size and the dataset are included in Appendix E.', but Appendix E is not provided. No explicit hyperparameters (e.g., learning rate, number of epochs, specific optimizer settings) or detailed training configurations are present in the main text. |