A Simple and Efficient Tensor Calculus
Authors: Sören Laue, Matthias Mitterreiter, Joachim Giesen4527-4534
AAAI 2020 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | For validating our framework we compute Hessians for several machine learning problems. It turns out that our method, because of the additional optimizations, outperforms the approach of (Laue, Mitterreiter, and Giesen 2018) which is already a few orders of magnitude more efficient than Tensor Flow, Py Torch, autograd, and JAX. We have implemented both modes of the tensor calculus from the previous section together with the improvements that can be achieved by cross-country mode and the compactification of higher order derivatives. We computed function values, gradients, and Hessians for each set of experiments. |
| Researcher Affiliation | Collaboration | S oren Laue,1,2 Matthias Mitterreiter,1 Joachim Giesen1 1Friedrich-Schiller-Universit at Jena Faculty of Mathematics and Computer Science Ernst-Abbe-Platz 2 07743 Jena, Germany Friedrich-Schiller-University Jena 2Data Assessment Solutions Gmb H, Hannover |
| Pseudocode | No | The paper describes the algorithms for forward and reverse mode automatic differentiation mathematically and textually, but it does not include any explicitly labeled pseudocode blocks or algorithms. |
| Open Source Code | Yes | An anonymized version of our implementation can be found in the supplemental material. |
| Open Datasets | No | To avoid the issue of sparsity patterns we generated dense, random data for each experiment. In this setting the running time does not depend on whether the data are synthetic or real world. |
| Dataset Splits | No | The paper describes generating dense, random data for its experiments and testing its approach on various machine learning problems (logistic regression, matrix factorization, neural net). However, it does not specify any training, validation, or test dataset splits or cross-validation methods. |
| Hardware Specification | Yes | The experiments were run in a pure CPU setting (Intel Xeon E5-2643, 8 cores) as well as in a pure GPU setting (NVIDIA Tesla V100), except for autograd that does not provide GPU support. |
| Software Dependencies | Yes | To evaluate expressions we used Num Py 1.16 and Cu Py 5.1. We compared our framework with the state-of-the-art automatic differentiation frameworks Tensor Flow 1.14, Py Torch 1.0, autograd 1.2, and JAX 0.1.27 used with Python 3.6 that were all linked against the Intel MKL. |
| Experiment Setup | Yes | We followed the experimental setup of (Laue, Mitterreiter, and Giesen 2018). However, we added one more experiment, a small neural net. We set m = 2n in the experiments. For the experiments, we set k = 5 and compute the gradient and Hessian with respect to U. We have created a small neural net with ten fully connected layers, Re LU activation functions, and a softmax cross-entropy output layer. |