Fast Differentiable Matrix Square Root
Authors: Yue Song, Nicu Sebe, Wei Wang
ICLR 2022 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Experimental results on the de-correlated batch normalization and second-order vision transformer demonstrate that our methods can also achieve competitive and even slightly better performances. |
| Researcher Affiliation | Academia | Yue Song, Nicu Sebe & Wei Wang Department of Information Engineering and Computer Science (DISI) University of Trento Trento, TN 38123, Italy {yue.song}@unitn.it |
| Pseudocode | Yes | Algorithm 1: FP of our MTP and MPA. Input: A and K Output: A 1 2 and Algorithm 2: BP of our Lyapunov solver. Input: l A 1 2 , A 1 2 , and T Output: l A |
| Open Source Code | Yes | The code is available at https://github.com/King James Song/Fast Differentiable Mat Sqrt. |
| Open Datasets | Yes | Experimental results on the de-correlated batch normalization and second-order vision transformer demonstrate that our methods can also achieve competitive and even slightly better performances. The code is available at https://github.com/King James Song/Fast Differentiable Mat Sqrt. ... Table 3 displays the speed and validation error on CIFAR10 and CIFAR100 (Krizhevsky, 2009). ... Table 4 compares the speed and performances on three So-Vi T architectures with different depths. ... Image Net (Deng et al., 2009). |
| Dataset Splits | Yes | Table 3 displays the speed and validation error on CIFAR10 and CIFAR100 (Krizhevsky, 2009). ... Table 4 compares the speed and performances on three So-Vi T architectures with different depths. ... Validation top-1/top-5 accuracy of the second-order vision transformer on Image Net (Deng et al., 2009). |
| Hardware Specification | Yes | The time consumption is measured for computing the matrix square root (BP+FP) on a workstation equipped with a Tesla K40 GPU and a 6-core Intel(R) Xeon(R) CPU @ 2.20GHz. ... The time cost is measured for computing the matrix square root (BP+FP) on a workstation equipped with a Tesla 4C GPU and a 6-core Intel(R) Xeon(R) CPU @ 2.20GHz. ... We use 8 Tesla G40 GPUs for distributed training and the NVIDIA Apex mixed-precision trainer is used. |
| Software Dependencies | No | All the source codes are implemented in Pytorch. ... the NVIDIA Apex mixed-precision trainer is used. The paper mentions PyTorch and NVIDIA Apex but does not provide specific version numbers for these software dependencies. |
| Experiment Setup | Yes | For our forward pass, we match the MTP to the power series of degree 11 and set the degree for both numerator and denominator of our MPA as 5. We keep iterating 8 times for our backward Lyapunov solver. ... we truncate the Taylor polynomial to degree 20 for SVD-Taylor. ... For the NS iteration, ... we set the iteration times to 5. |