Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in [1].
GluonCV and GluonNLP: Deep Learning in Computer Vision and Natural Language Processing
Authors: Jian Guo, He He, Tong He, Leonard Lausen, Mu Li, Haibin Lin, Xingjian Shi, Chenguang Wang, Junyuan Xie, Sheng Zha, Aston Zhang, Hang Zhang, Zhi Zhang, Zhongyue Zhang, Shuai Zheng, Yi Zhu
JMLR 2020 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We demonstrate the performance of Gluon CV/NLP models in various computer vision and natural language processing tasks. Specifically, we evaluate popular or state-of-the-art models on standard benchmark data sets. In the experiments, we compare model performance between Gluon CV/NLP and other open source implementations with Caffe, Caffe2, Theano, and Tensor Flow, including Res Net (He et al., 2016) and Mobile Net (Howard et al., 2017) for image classification (Image Net), Faster R-CNN (Girshick, 2015) for object detection (COCO), Mask R-CNN (He et al., 2017) for instance segmentation, Simple Pose (Xiao et al., 2018) for pose estimation (COCO), Inflated 3D networks (I3D) (Carreira and Zisserman, 2017) for video action recognition, text CNN (Kim, 2014) for sentiment analysis (TREC), and BERT (Devlin et al., 2018) for question answering (SQu AD 1.1), sentiment analysis (SST-2), natural langauge inference (MNLI-m), and paraphrasing (MRPC). Table 1 shows that the Gluon CV/Gluon NLP implementation matches or outperforms the compared open source implementation for the same model evaluated on the same data set. |
| Researcher Affiliation | Collaboration | Jian Guo EMAIL University of Michigan MI, USA He He EMAIL Tong He EMAIL Leonard Lausen EMAIL Mu Li EMAIL Haibin Lin EMAIL Xingjian Shi EMAIL Chenguang Wang EMAIL Junyuan Xie EMAIL Sheng Zha EMAIL Aston Zhang EMAIL Hang Zhang EMAIL Zhi Zhang EMAIL Zhongyue Zhang EMAIL Shuai Zheng EMAIL Yi Zhu EMAIL Amazon Web Services, CA, USA |
| Pseudocode | No | The paper includes a Python code snippet demonstrating API usage, not a generic algorithm or pseudocode. The text is: 'The following code snippet shows an example of using fixed bucketing to sample minibatches more efficiently. In the first assignment statement, the batchify utilities provided by the data API specifies that inputs of different shapes will be padded to have the same length then be stacked as a mini-batch. The following Fixed Bucket Sampler class of the data API will group inputs of more similar shapes into the same mini-batch so more computation is saved due to less padding. In the end, we pass the aforementioned specifications, gluon , gluonnlp batchify_fn = gluonnlp.data.batchify.Tuple(gluonnlp.data.batchify.Pad(), gluonnlp.data.batchify.Stack ()) train_sampler = gluonnlp.data. Fixed Bucket Sampler (lengths=train_data.transform(lambda x: len(x[0])), batch_size=batch_size , shuffle=True) train_iter = gluon.data.Data Loader(train_data , batchify_fn=batchify_fn , batch_sampler = train_sampler )' |
| Open Source Code | Yes | We present Gluon CV and Gluon NLP, the deep learning toolkits for computer vision and natural language processing based on Apache MXNet (incubating). These toolkits provide state-of-the-art pre-trained models, training scripts, and training logs, to facilitate rapid prototyping and promote reproducible research. We also provide modular APIs with flexible building blocks to enable efficient customization. Leveraging the MXNet ecosystem, the deep learning models in Gluon CV and Gluon NLP can be deployed onto a variety of platforms with different programming languages. The Apache 2.0 license has been adopted by Gluon CV and Gluon NLP to allow for software distribution, modification, and usage. The documentation https://gluon-cv.mxnet.io/ and http://gluon-nlp.mxnet.io/ of Gluon CV/NLP include installation instructions, contribution instructions, open source repositories, extensive API reference, and comprehensive tutorials. |
| Open Datasets | Yes | Besides, users can access a wide range of popular data sets via the data API, including (but are not limited to) Image Net of image classification, VOC of object detection, COCO of instance segmentation, SST of sentiment analysis, IWSLT of machine translation, SQu AD of question answering, and Wiki Text of language modeling. The code snippet below shows that users can access training sets of IWSLT2015 English-Vietnamese and SQu AD 2.0, and the test set of Wiki Text103 with just one line of code via the data API. import gluonnlp iwslt15 = gluonnlp.data.IWSLT2015( train , src_lang= en , tgt_lang= vi ) squad = gluonnlp.data.SQu AD( train , 2.0 ) wikitext103 = gluonnlp.data.Wiki Text103( test ) |
| Dataset Splits | Yes | Table 1: Comparison of model performance (in percentage) on the validation data sets between Gluon CV/NLP and other open source implementations (OOSI) across popular computer vision and natural language processing tasks and data sets. Also: import gluonnlp iwslt15 = gluonnlp.data.IWSLT2015( train , src_lang= en , tgt_lang= vi ) squad = gluonnlp.data.SQu AD( train , 2.0 ) wikitext103 = gluonnlp.data.Wiki Text103( test ) |
| Hardware Specification | Yes | Figure 1 visualizes inference throughputs (on 1 NVIDIA TESLA V100) vs. validation accuracy. Circle Area is proportional to device memory requirement. Also: comparing with the float32 inference, our sentence classification result on the MRPC dataset shows that the int8 inference reduces latency of the BERTBASE model by 59.6% on Intel CLX-8280. |
| Software Dependencies | No | The paper mentions 'Gluon CV/NLP are implemented in Python' and refers to 'minimum and open source package (e.g., MXNet) requirements are specified in the documentation.' However, it does not provide specific version numbers for these software components in the main text. |
| Experiment Setup | No | The paper states: 'We highlight that the outperformance can be attributed to minor refinements such as in data augmentations and optimization methods in the training procedure (He et al., 2019).' However, it does not provide specific hyperparameter values, detailed training configurations, or system-level settings for the experiments in the main text. |