DiffChaser: Detecting Disagreements for Deep Neural Networks
Authors: Xiaofei Xie, Lei Ma, Haijun Wang, Yuekang Li, Yang Liu, Xiaohong Li
IJCAI 2019 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | To demonstrate the effectiveness and usefulness of Diff Chaser, we evaluate and compare it with the state-of-the-art techniques. The results confirm that Diff Chaser can generate much more disagreements efficiently. We further apply Diff Chaser on real products, i.e., Tensor Flow Lite and Core ML, the results demonstrate that Diff Chaser can generate disagreements with a high success rate. |
| Researcher Affiliation | Academia | Xiaofei Xie1 , Lei Ma2 , Haijun Wang1 , Yuekang Li1 , Yang Liu1,3 and Xiaohong Li4 Nanyang Technological University, Singapore 2Kyushu University, Japan 3Zhejiang Sci-Tech University, China 4Tianjin University, China |
| Pseudocode | Yes | Algorithm 1: Disagreement generation input : x: an input, f1 and f2: two DNN models output: xd: a disagreement example const : m: size of population, r2: mutation rate 1 Construct an initial population X from input x; 2 iteration := 0; 3 while True do 4 if timeout or exceed the maximum iterations then 5 return failed; 6 Calculate fitness values for the chromosomes in X; 7 if xd X such that xd is a disagreement example then 8 return xd; 9 for i [0, m) do 10 if X[i] has the best fitness value then 11 continue; 12 Select two chromosomes x1 and x2 from X; 13 X[i] := crossover(x1, x2); 14 X[i] := mutate(X[i], r2); 15 iteration+ = 1; |
| Open Source Code | No | The paper states 'We have implemented Diff Chaser in Python based on Keras (ver.2.1.3) with Tensor Flow (ver.1.5.0) backend.' However, it does not provide any link or explicit statement about making the source code for Diff Chaser publicly available. |
| Open Datasets | Yes | We select two popular publicly available datasets (MNIST [Le Cun and Cortes, 1998] and CIFAR10 [Krizhevsky et al., 2014]) as the evaluation subjects. |
| Dataset Splits | No | The paper mentions selecting 'seed inputs from the test data' but does not provide specific details on training, validation, or test dataset splits. It does not mention a validation set at all. |
| Hardware Specification | Yes | All the experiments were run on a server with the Ubuntu 16.04 system with 28-core 2.0GHz Xeon CPU, 196 GB RAM and 4 NVIDIA Tesla V100 16G GPUs. |
| Software Dependencies | Yes | We have implemented Diff Chaser in Python based on Keras (ver.2.1.3) with Tensor Flow (ver.1.5.0) backend. |
| Experiment Setup | Yes | In the evaluation, the size of the population is set to 1,000. For each configuration, the maximum iteration (see Algorithm 1) is set to 100. We set the maximum iterations to 300... |