Differentiable Optimization of Generalized Nondecomposable Functions using Linear Programs
Authors: Zihang Meng, Lopamudra Mukherjee, Yichao Wu, Vikas Singh, Sathya Ravi
NeurIPS 2021 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | In this section, we conduct experiments on commonly used benchmarks to show that our framework can be used to optimize multiple different objectives within deep neural networks and lead to performance gain. We start with binary AUC optimization, and then extend to multi class AUC optimization and F-score optimization. |
| Researcher Affiliation | Academia | 1University of Wisconsin-Madison 2University of Wisconsin-Whitewater 3University of Illinois at Chicago |
| Pseudocode | Yes | Figure 1: Overview of how to compute differentiable non-decomposable metric and code example of our LP solver. We provide sparse support for operations which are not supported in official Pytorch library (colored operations in the figure). [Figure 1 contains a numbered code-like block for the LP solver] |
| Open Source Code | Yes | Our code is available at https://github.com/zihangm/nondecomposable-as-lp. |
| Open Datasets | Yes | Datasets: Cat&Dog, CIFAR10, CIFAR100, and STL10 (See appendix for dataset details). |
| Dataset Splits | Yes | We follow Liu et al. [2019] to use 19k/1k, 45k/5k, 45k/5k, 4k/1k training/validation split on Cat&Dog, CIFAR10, CIFAR100, STL10 respectively. |
| Hardware Specification | Yes | On the other hand, when the matrix is sparse, with our new sparse implementation, we show that we can train Resnet-18 with 10 mini-batch size (memory savings) on a 2080TI Nvidia GPU. |
| Software Dependencies | No | Our approach is based on the premise that tackling the LP form of the non-decomposable objective as a module within the DNN, one which permits forward and reverse mode differentiation and can utilize in-built support for specialized GPU hardware in modern libraries such as Py Torch, is desirable. The paper mentions software such as PyTorch, CVXPY, and scipy but does not provide specific version numbers for any of them. |
| Experiment Setup | Yes | We use the same random seed, learning rate and total number of iterations in all of our experiments including multi class AUC and F-score experiments. See appendix for model architecture, learning rate, etc. |