Accelerating Feedforward Computation via Parallel Nonlinear Equation Solving
Authors: Yang Song, Chenlin Meng, Renjie Liao, Stefano Ermon
ICML 2021 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Experimentally, we demonstrate the effectiveness of our approach in accelerating (i) backpropagation of RNNs, (ii) evaluation of Dense Nets, and (iii) autoregressive sampling of MADE and Pixel CNN++, with speedup factors between 2.1 and 26 under various settings. |
| Researcher Affiliation | Academia | 1Computer Science Department, Stanford University. 2Department of Computer Science, University of Toronto. 3Vector Institute. |
| Pseudocode | Yes | Algorithm 1 Nonlinear Jacobi Iteration, Algorithm 2 Nonlinear Jacobi-GS Iteration, Algorithm 3 Nonlinear GS-Jacobi Iteration |
| Open Source Code | No | The paper does not provide a direct link to its source code or explicitly state that its code is open-source. |
| Open Datasets | Yes | We compared Jacobi iteration against feedforward sampling for models trained on MNIST (Le Cun & Cortes, 2010) and CIFAR-10 (Krizhevsky et al., 2009) respectively. [...] We use a Dense Net-201 model pre-trained on Image Net (Russakovsky et al., 2015). |
| Dataset Splits | No | The paper mentions using well-known datasets and a synthesized dataset from MNIST, but does not explicitly specify training/validation/test splits for its own experiments, or refer to standard predefined splits for the specific models it evaluates. |
| Hardware Specification | Yes | measured with wall-clock time on a single GPU. [...] measured in actual wallclock time on a single NVIDIA Titan Xp GPU [...] measured on a single NVIDIA Tesla V100 GPU with 32 GB memory. |
| Software Dependencies | No | The paper mentions deep learning frameworks like Py Torch (Paszke et al., 2019), Tensor Flow (Abadi et al., 2015), and JAX (Bradbury et al., 2018), but does not provide specific version numbers for these or other software used in their experiments. |
| Experiment Setup | Yes | We train a simple RNN with one hidden layer to fit sequences. [...] We use a Dense Net-201 model pre-trained on Image Net. [...] For GS-Jacobi, one block contains 15 rows of pixels on MNIST and 2 rows of pixels on CIFAR-10. For Jacobi-GS, one block has one row of pixels on both datasets. [...] The batch sizes are 16 and 4 for MNIST and CIFAR-10 respectively. [...] stop the iteration once the forward difference (defined in Section 2) is below a chosen tolerance value ϵ. |