RS-Del: Edit Distance Robustness Certificates for Sequence Classifiers via Randomized Deletion
Authors: Zhuoqun Huang, Neil G Marchant, Keane Lucas, Lujo Bauer, Olga Ohrimenko, Benjamin Rubinstein
NeurIPS 2023 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Finally, we present a comprehensive evaluation of RS-Del in the malware analysis setting.1 We investigate tradeoffs between accuracy and the size of our certificates for varying levels of deletion, and observe that RS-Del can achieve a certified accuracy of 91% at an edit distance radius of 128 bytes using on the order of 103 model queries. |
| Researcher Affiliation | Academia | Zhuoqun Huang University of Melbourne zhuoqun@unimelb.edu.au Neil G. Marchant University of Melbourne nmarchant@unimelb.edu.au Keane Lucas Carnegie Mellon University keanelucas@cmu.edu Lujo Bauer Carnegie Mellon University lbauer@cmu.edu Olga Ohrimenko University of Melbourne oohrimenko@unimelb.edu.au Benjamin I. P. Rubinstein University of Melbourne brubinstein@unimelb.edu.au |
| Pseudocode | Yes | Figure 1: Probabilistic certification of RS-Del. Here x is the input sequence, fb is the base classifier, pdel is the deletion probability, η is the set of decision thresholds, α is the significance level, and npred, nbnd are sample sizes. Bin LCB(k, n, α) returns a lower confidence bound for p at level α given k Bin(n, p). |
| Open Source Code | Yes | Our implementation is available at https://github.com/Dovermore/randomized-deletion. |
| Open Datasets | Yes | We use two Windows malware datasets: Sleipnir2 which is compiled from public sources following Al-Dujaili et al. [54] and VTFeed which is collected from Virus Total [17]. |
| Dataset Splits | Yes | Table 3: Summary of datasets. Number of samples Dataset Label Train Validation Test Sleipnir2 Benign 20 948 7 012 6 999 Malicious 20 768 6 892 6 905 VTFeed Benign 111 258 13 961 13 926 Malicious 111 395 13 870 13 906. The dataset is randomly split into training, validation and test sets with a ratio of 60%, 20% and 20% respectively. Following Lucas et al. [17], the dataset is randomly split into training, validation and test sets with a ratio of 80%, 10%, and 10% respectively. |
| Hardware Specification | Yes | Table 4: Compute resources used for training. ... Sleipnir2 1 NVIDIA P100 GPU, 4 cores on Intel Xeon Gold 6326 CPU ... VTFeed 1 NVIDIA RTX3090 GPU, 6 cores on AMD Ryzen Threadripper PRO 3975WX CPU |
| Software Dependencies | No | All times are recorded on a desktop PC fitted with an AMD Ryzen 7 5800X CPU and an NVIDIA RTX3090 GPU, using our Py Torch implementation of RS-Del and RS-Abn. |
| Experiment Setup | Yes | Table 6: Parameter settings for Mal Conv, the optimizer and training procedure. Parameter Values Max input size 2097152 Embedding size 8 Window size 500 Channels 128 Python class torch.optim.SGD Learning rate 0.01 Momentum 0.9 Weight decay 0.001 Batch size 24 (Sleipnir2), 32 (VTFeed) Max. epoch 50 (Sleipnir2), 100 (VTFeed) Min. preserved bytes 500 (RS-Del, RS-Abn), NA (NS) Embedding gradient clipping 0.5 (RS-Abn), (RS-Del, NS) Early stopping If validation loss does not improve after 10 epochs |