The CLRS Algorithmic Reasoning Benchmark
Authors: Petar Veličković, Adrià Puigdomènech Badia, David Budden, Razvan Pascanu, Andrea Banino, Misha Dashevskiy, Raia Hadsell, Charles Blundell
ICML 2022 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We perform extensive experiments to demonstrate how several popular algorithmic reasoning baselines perform on these tasks, and consequently, highlight links to several open challenges. |
| Researcher Affiliation | Industry | 1Deep Mind. |
| Pseudocode | Yes | In order to further illustrate how these hints are collected, we also provide an informal pseudocode for collecting hints for insertion sort in Algorithm 1: |
| Open Source Code | Yes | Our library is readily available at https://github.com/deepmind/clrs. The CLRS benchmark is publicly hosted on Git Hub: https://github.com/deepmind/clrs. All code and artifacts are released under an Apache 2.0 license, which is highly permissive. |
| Open Datasets | Yes | Our library is readily available at https://github.com/deepmind/clrs. For each algorithm in CLRS-30, we provide a canonical set of training, validation and test trajectories for benchmarking inand out-of-distribution generalisation. We obtain these trajectories by running the algorithms on randomly sampled inputs that conform to their input specification. |
| Dataset Splits | Yes | For validation, our aim is to measure in-distribution generalisation. Hence we sample inputs of 16 nodes for both, and generate 1,000 trajectories for training and 32 for validation. |
| Hardware Specification | Yes | Our models are trained on one V100 Volta GPU, requiring roughly between 1h and 30h to train, depending on the algorithm s time complexity. |
| Software Dependencies | No | The paper mentions software like JAX, Haiku, NumPy, PyTorch, and TensorFlow, but it does not specify version numbers for these dependencies (e.g., 'Our baselines are provided in JAX and Haiku (Hennigan et al., 2020), but the dataset is generated using Num Py, making it possible to create learning pipelines in virtually any framework, including Py Torch and Tensor Flow.'). |
| Experiment Setup | Yes | We optimise our models on the training trajectories in a teacher-forced fashion, with a batch size of 32, using the Adam optimiser (Kingma & Ba, 2014) with an initial learning rate of η = 0.001. We train for 10, 000 steps, early stopping on the validation performance. |