On Correctness of Automatic Differentiation for Non-Differentiable Functions
Authors: Wonyeol Lee, Hangyeol Yu, Xavier Rival, Hongseok Yang
NeurIPS 2020 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Theoretical | In this paper, we provide a positive answer to this question. Using counterexamples, we first point out flaws in oftenused informal arguments, such as: non-differentiabilities arising in deep learning do not cause any issues because they form a measure-zero set. We then investigate a class of functions, called PAP functions, that includes nearly all (possibly nondifferentiable) functions in deep learning nowadays. For these PAP functions, we propose a new type of derivatives, called intensional derivatives, and prove that these derivatives always exist and coincide with standard derivatives for almost all inputs. We also show that these intensional derivatives are what most autodiff systems compute or try to compute essentially. In this way, we formally establish the correctness of autodiff systems applied to non-differentiable functions. |
| Researcher Affiliation | Academia | Wonyeol Lee Hangyeol Yu Xavier Rival Hongseok Yang School of Computer Science, KAIST, South Korea INRIA Paris, Département d Informatique of ENS, and CNRS/PSL University, France {wonyeol.lee.cs, yhk1344}@gmail.com rival@di.ens.fr hongseok.yang@kaist.ac.kr |
| Pseudocode | No | The paper does not contain any sections or figures explicitly labeled as 'Pseudocode' or 'Algorithm'. |
| Open Source Code | No | The paper mentions popular autodiff systems like Tensor Flow and Py Torch but does not provide a link or statement about releasing its own source code for the methodology described. |
| Open Datasets | No | This is a theoretical paper and does not involve empirical studies with datasets for training. |
| Dataset Splits | No | This is a theoretical paper and does not involve empirical studies that would require training/validation/test dataset splits. |
| Hardware Specification | No | This is a theoretical paper and does not describe any specific hardware used for experiments. |
| Software Dependencies | No | The paper mentions existing autodiff systems like TensorFlow and PyTorch, but it does not specify any software dependencies with version numbers for its own theoretical work. |
| Experiment Setup | No | This is a theoretical paper and does not describe experimental setup details such as hyperparameters or system-level training settings. |