Confusion Graph: Detecting Confusion Communities in Large Scale Image Classification
Authors: Ruochun Jin, Yong Dou, Yueqing Wang, Xin Niu
IJCAI 2017 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Utilizing this community information, we can also employ pre-trained models to automatically identify mislabeled images in the large scale database. With our method, researchers just need to manually check approximate 3% of the ILSVRC2012 classification database to locate almost all mislabeled samples. For illustration purpose, we select ten 3-class communities and design specialized layers to overcome each weakness. For Alex Netbased and VGG-verydeep-16-based models, the mean decreases of the top-1 error rate are 1.49% and 3.45% respectively, which are comparable to other state-of-the-art methods [Yan et al., 2015; Ahmed et al., 2016]. Secondly, we employ pre-trained models along with the community information to automatically identify mislabeled samples in the image database. Evaluated with the randomly polluted Oxford102 flowers dataset where 15% of the images are mislabeled, our method can detect approximate 89% of all wrong labels with the precision of 72%. |
| Researcher Affiliation | Academia | Ruochun Jin, Yong Dou, Yueqing Wang and Xin Niu National Laboratory for Parallel and Distributed Processing, National University of Defense Technology, Changsha, Hunan, 410073, China {jinruochun,yongdou,xinniu}@nudt.edu.cn, yqwang2013@163.com |
| Pseudocode | Yes | Algorithm 1: Establish the confusion graph of a N-class classification |
| Open Source Code | No | No explicit statement or link providing concrete access to the source code for the methodology described in this paper was found. The paper mentions using "Matlab2014a with Mat Conv Net [Vedaldi and Lenc, 2015] and the pretrained models are downloaded from the Mat Conv Net website", which refers to external resources used, not their own code. |
| Open Datasets | Yes | large scale image datasets such as Image Net [Deng et al., 2009]. Le Net [Le Cun et al., 1995] with the CIFAR10 [Krizhevsky and Hinton, 2009] validation set. CIFAR100 [Krizhevsky and Hinton, 2009]. Oxford 102 flowers dataset [Nilsback and Zisserman, 2008]. |
| Dataset Splits | Yes | we evaluate the pre-trained Le Net [Le Cun et al., 1995] with the CIFAR10 [Krizhevsky and Hinton, 2009] validation set. We train each ES with images of the corresponding 3 classes from the ILSVRC2012 training set. We tested each refined model using images from the ILSVRC2012 validation set. Then we use the validation set, containing 1020 images, i.e. 10 image for each class, to construct the confusion graph and obtain the community list of each model. |
| Hardware Specification | No | No specific hardware details (e.g., GPU/CPU models, processor types, memory amounts) used for running the experiments were provided. The paper only mentions software used: "All experiments in this paper are completed using Matlab2014a with Mat Conv Net". |
| Software Dependencies | Yes | All experiments in this paper are completed using Matlab2014a with Mat Conv Net [Vedaldi and Lenc, 2015] |
| Experiment Setup | Yes | Using Algorithm 1 with τ set as 5, for illustration purpose, we evaluate the pre-trained Le Net. For each selected community, we train an Alex Net-based expert sub-net (ES) which is shown in Figure 5. Each ES contains three full connection layers and the forward prediction process can be divided into 2 stages. The parameter µ controls the trade-off between the precision and the recall of the detection, where a high µ leads to high precision with low recall and a low µ leads to low precision with high recall. In practice, we iteratively correct the mislabeled images by manually checking the output of Algorithm 2 in each iteration. The parameter µ decreases from 5 to 2 successively during iterations of auto-detection and the iterative process ends until no image in the output is true mislabeled. |