Open-Book Neural Algorithmic Reasoning
Authors: Hefei Li, Peng Chao, Chenyang Xu, Zhengfeng Yang
NeurIPS 2024 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Empirical evaluation is conducted on the challenging CLRS Algorithmic Reasoning Benchmark, which consists of 30 diverse algorithmic tasks. Our open-book learning framework exhibits a significant enhancement in neural reasoning capabilities. |
| Researcher Affiliation | Academia | Hefei Li, Chao Peng , Chenyang Xu , Zhengfeng Yang Shanghai Key Laboratory of Trustworthy Computing Software Engineering Institute East China Normal University, Shanghai, China 51255902127@stu.ecnu.edu.cn, {cpeng, cyxu, zfyang}@sei.ecnu.edu.cn |
| Pseudocode | Yes | Algorithm 1 Attention-Based Implementation of Open-Book Reasoning |
| Open Source Code | Yes | The codes are provided in https://github.com/Hoferlee1/Open-Book |
| Open Datasets | Yes | Challenging Benchmark for NAR. CLRS Algorithmic Reasoning Benchmark proposed by [26] is currently the most popular and definitive benchmark for evaluating the algorithmic capabilities of neural networks. |
| Dataset Splits | No | The paper mentions 'training and test sets' but does not specify the explicit split percentages, sample counts, or methodology for creating validation splits for its experiments. It notes that 'The test instances are substantially larger in scale compared to those in the training set' which describes a characteristic of the benchmark, not the explicit split used. |
| Hardware Specification | Yes | The experiments are conducted on a machine equipped with an i7-13700K CPU, an RTX 4090 GPU, and an RTX A6000 GPU. |
| Software Dependencies | No | The paper does not list specific version numbers for any software dependencies, libraries, or frameworks used in the experiments. |
| Experiment Setup | Yes | To ensure fair comparisons, we follow the widely-used experimental hyperparameter settings in [11], where the batch size is 32 and the network is trained for 10,000 steps by Adam optimizer with a learning rate of 0.001. During each training and testing iteration, we allow Algorithm 1 to sample 240 auxiliary data points and use only one attention head. |