Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in Coakley et alK. L. Coakley, T. Snelleman, H. Hoos, and O. E. Gundersen, "The embrace of open science: An analysis of a decade of AI research and 56 800 conference papers," Under Review, 2026..
Candidates vs. Noises Estimation for Large Multi-Class Classification Problem
Authors: Lei Han, Yiheng Huang, Tong Zhang
ICML 2018 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Extensive experimental results show that CANE achieves better prediction accuracy over the Noise-Contrastive Estimation (NCE), its variants and a number of the state-of-the-art tree classifiers, while it gains significant speedup compared to standard O(K) methods. We evaluate the CANE method in various applications in this section, including both multi-class classification problems and neural language modeling. |
| Researcher Affiliation | Industry | 1Tencent AI Lab, Shenzhen, China. Correspondence to: Lei Han <EMAIL>, Yiheng Huang <EMAIL>, Tong Zhang <EMAIL>. |
| Pseudocode | Yes | Algorithm 1 A general optimization procedure for CANE. Algorithm 2 The Beam Tree Algorithm. |
| Open Source Code | No | The paper does not provide an explicit statement or link for the open-sourcing of their own code. It only references an external platform (Vowpal-Wabbit) used for comparison, with a link to that platform's repository. |
| Open Datasets | Yes | We consider four multi-class classification problems, including the Sector3 dataset with 105 classes (Chang & Lin, 2011), the ALOI4 dataset with 1000 classes (Geusebroek et al., 2005), the Image Net-20105 dataset with 1000 classes, and the Image Net-10K5 dataset with 10K classes (Image Net Fall 2009 release). We test the methods on two benchmark corpora: the Penn Tree Bank (PTB) (Mikolov et al., 2010) and Gutenberg8 corpora. |
| Dataset Splits | Yes | The data from Sector and ALOI is split into 90% training and 10% testing. In Image Net-2010, the training set contains 1.3M images and we use the validation set containing 50K images as the test set. The Image Net-10K data contains 9M images and we randomly split the data into two halves for training and testing by following the protocols in (Deng et al., 2010; S anchez & Perronnin, 2011; Le, 2013). |
| Hardware Specification | Yes | All the methods are implemented using a standard CPU machine with quad-core Intel Core i5 processor. The experiments in this section are implemented on a machine with NVIDIA Tesla M40 GPUs. |
| Software Dependencies | No | The paper mentions software like 'Vowpal-Wabbit' and 'VGG-16 net' but does not specify any version numbers for these or any other software dependencies. |
| Experiment Setup | Yes | We use b-nary tree for CANE and set b = 10 for all classification problems. We set k = 10 for Sector and ALOI and k = 20 for Image Net-2010 and Image Net-10K. All the methods use SGD with learning rate selected from {0.0001, 0.001, 0.01, 0.05, 0.1, 0.5, 1.0}. We run all the methods 50 epochs on Sector, ALOI and Image Net-2010 datasets and 20 epochs on Image Net-10K. We set the embedding size as 256 and use a LSTM model with 512 hidden states and 256 projection size. The sequence length is fixed as 20 and the learning rate is selected from {0.025, 0.05, 0.1, 0.2}. |