Learning with Symmetric Label Noise: The Importance of Being Unhinged
Authors: Brendan van Rooyen, Aditya Menon, Robert C. Williamson
NeurIPS 2015 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We now illustrate that the unhinged loss SLN-robustness is empirically manifest. We reiterate that with high regularisation, the unhinged solution is equivalent to an SVM (and in the limit any classification-calibrated loss) solution. Thus, we do not aim to assert that the unhinged loss is better than other losses, but rather, to demonstrate that its SLN-robustness is not purely theoretical. We first show that the unhinged risk minimiser performs well on the example of Long and Servedio [2010] (henceforth LS10). Figure 1 shows the distribution D, where X = {(1, 0), (γ, 5γ), (γ, γ)} R2, with marginal distribution M = { 1 2} and all three instances are deterministically positive. We pick γ = 1/2. The unhinged minimiser perfectly classifies all three points, regardless of the level of label noise (Figure 1). The hinge minimiser is perfect when there is no noise, but with even a small amount of noise, achieves a 50% error rate. We next consider empirical risk minimisers from a random training sample: we construct a training set of 800 instances, injected with varying levels of label noise, and evaluate classification performance on a test set of 1000 instances. We compare the hinge, t-logistic (for t = 2) [Ding and Vishwanathan, 2010] and unhinged minimisers using a linear scorer without a bias term, and regularisation strength λ = 10 16. From Table 1, even at 40% label noise, the unhinged classifier is able to find a perfect solution. By contrast, both other losses suffer at even moderate noise rates. We next report results on some UCI datasets, where we additionally tune a threshold so as to ensure the best training set 0-1 accuracy. Table 2 summarises results on a sample of four datasets. (The Appendix contains results with more datasets, performance metrics, and losses.) Even at noise close to 50%, the unhinged loss is often able to learn a classifier with some discriminative power. |
| Researcher Affiliation | Collaboration | Brendan van Rooyen , Aditya Krishna Menon , Robert C. Williamson , The Australian National University National ICT Australia { brendan.vanrooyen, aditya.menon, bob.williamson }@nicta.com.au |
| Pseudocode | No | The paper does not contain any pseudocode or clearly labeled algorithm blocks. |
| Open Source Code | No | The paper does not provide an unambiguous statement or link regarding the release of source code for the methodology described. |
| Open Datasets | Yes | We next report results on some UCI datasets, where we additionally tune a threshold so as to ensure the best training set 0-1 accuracy. Table 2 summarises results on a sample of four datasets. (The Appendix contains results with more datasets, performance metrics, and losses.) |
| Dataset Splits | No | The paper mentions training and testing sets, and tuning on the training set, but does not explicitly define a separate validation set or split percentages for validation. |
| Hardware Specification | No | The paper does not provide specific details about the hardware used to run the experiments. |
| Software Dependencies | No | The paper does not specify any software dependencies with version numbers needed to replicate the experiments. |
| Experiment Setup | Yes | We compare the hinge, t-logistic (for t = 2) [Ding and Vishwanathan, 2010] and unhinged minimisers using a linear scorer without a bias term, and regularisation strength λ = 10 16. From Table 1, even at 40% label noise, the unhinged classifier is able to find a perfect solution. By contrast, both other losses suffer at even moderate noise rates. We next report results on some UCI datasets, where we additionally tune a threshold so as to ensure the best training set 0-1 accuracy. |