Adversarial Robustness for Code
Authors: Pavol Bielik, Martin Vechev
ICML 2020 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We instantiated our approach to a task studied by a number of prior works predicting types for two dynamically typed languages Java Script and Type Script (Hellendoorn et al., 2018; Schrouff et al., 2019; Malik et al., 2019; Raychev et al., 2015). In this task, the need for model robustness is natural since the model is queried each time a program is modifed by the user. Our key results are: Our approach learns accurate and adversarially robust models for the task of type inference, achieving 87.7% accuracy while improving robustness from 52.1% to 67.0%. |
| Researcher Affiliation | Academia | Pavol Bielik 1 Martin Vechev 1 1Department of Computer Science, ETH Z urich, Switzerland. |
| Pseudocode | Yes | Algorithm 1 Training procedure used to learn a single adversarially robust model hf, gh, αi. ... Algorithm 2 Training multiple adversarially robust models, each of which learns to make predictions for a different subset of the dataset D. |
| Open Source Code | Yes | The code and datasets are available at: https://github.com/eth-sri/robust-code |
| Open Datasets | Yes | For our dataset, we collect the same top starred projects on Github and perform similar preprocessing steps as Hellendoorn et al. We provide detailed description in the supplementary material. The code and datasets are available at: https://github.com/eth-sri/robust-code |
| Dataset Splits | No | The paper mentions a 'training dataset D' and evaluates on a 'test set', but does not explicitly state the proportions or counts for training, validation, and test splits needed to reproduce data partitioning. The term 'validation' is not used for data splits. |
| Hardware Specification | Yes | We used a single Nvidia TITAN RTX for all the experiments. |
| Software Dependencies | No | We implemented our code in Py Torch (Paszke et al., 2019) and DGL library (Wang et al., 2019). |
| Experiment Setup | Yes | All models were trained with an embedding and hidden size of 128, batch size of 32, dropout 0.1 (Srivastava et al., 2014), initial learning rate of 0.001, using Adam opti mizer (Kingma & Ba, 2014) and between 10 to 20 epochs. |