POPQORN: Quantifying Robustness of Recurrent Neural Networks
Authors: Ching-Yun Ko, Zhaoyang Lyu, Lily Weng, Luca Daniel, Ngai Wong, Dahua Lin
ICML 2019 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We demonstrate its effectiveness on different network architectures and show that the robustness quantification on individual steps can lead to new insights. Experiments Methods. The proposed POPQORN is used to quantify robustness for all models herein. We implement our algorithm using Py Torch to enable the use of GPUs. |
| Researcher Affiliation | Academia | 1The University of Hong Kong, Hong Kong 2The Chinese University of Hong Kong, Hong Kong 3Massachusetts Institute of Technology, Cambridge, MA, USA. |
| Pseudocode | Yes | Proposed algorithms are summarized in Section A.2 and A.5 in the appendix. Figure 4. Steps in computing bounds for RNN networks. Input: initialize ϵ:= ϵ0, trained RNN, test sequence X, threshold τ |
| Open Source Code | Yes | Source code is available at https://github.com/Zhaoyang Lyu/POPQORN. |
| Open Datasets | Yes | In this experiment, we evaluate POPQORN and other baselines on totally 12 vanilla RNNs and LSTMs trained on the MNIST dataset. Next, we evaluate the proposed POPQORN quantification on LSTMs trained on the MNIST sequence dataset. Lastly, POPQORN is evaluated on LSTM classifiers for the question classification (UIUC’s Cog Comp QC) Dataset (Li & Roth, 2002). |
| Dataset Splits | No | The paper mentions using specific datasets like MNIST, but it does not provide details on training, validation, or test splits (e.g., percentages or sample counts). |
| Hardware Specification | Yes | Using a server consisting of 16 NVIDIA Tesla K80 GPUs, it took about 4 hours to calculate certified bounds for 1000 samples in LSTMs with 4 frames, and one day in LSTMs with 14 frames. Except quantifying for LSTMs, the remaining experiments could be finished in about 10 hours using a single NVIDIA TITAN X GPU. |
| Software Dependencies | No | We implement our algorithm using Py Torch to enable the use of GPUs. The paper mentions PyTorch but does not provide a specific version number for it or other software dependencies. |
| Experiment Setup | Yes | In practice, we start by initializing the ϵ to be ϵ0 > 0 and deriving upper and lower bounds according to Table 4. These are followed by a condition check that governs the next step. If i = j such that γU i γL j , then we decrease ϵ; otherwise, we increase ϵ. We repeat the above procedure until a predefined maximum iterations are reached or when convergence is achieved.2 Proposed algorithms are summarized in Section A.2 and A.5 in the appendix. Figure 4. Steps in computing bounds for RNN networks. Input: initialize ϵ:= ϵ0, trained RNN, test sequence X, threshold τ. |