A Model-Agnostic Heuristics for Selective Classification
Authors: Andrea Pugnana, Salvatore Ruggieri
AAAI 2023 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Experiments on real-world data show that SCROSS improves over existing methods. |
| Researcher Affiliation | Academia | 1 Scuola Normale Superiore, Pisa, Italy 2 University of Pisa, Pisa, Italy |
| Pseudocode | Yes | Algorithm 1: SCROSS.fit() |
| Open Source Code | Yes | Python source code and experimental notebooks are available at https://github.com/andrepugni/SCross. |
| Open Datasets | Yes | Adult. We dropped from the raw census data (Dua and Graff 2017) the instances with missing values... Lending. The Lending Club dataset3 regards loans in an online platform. (footnote 3: https://www.kaggle.com/wordsforthewise/lending-club) ... UCICredit. This dataset from (Dua and Graff 2017) concerns whether or not a credit card holder will default in the next six months (Yeh and Lien 2009). |
| Dataset Splits | Yes | The validation sets for SELNET and SAT consist of 10% of the training set and 2,000 instances, respectively, as in their original implementations... The split into training and test set is time-based when a timestamp feature is available and stratified random otherwise. ... The final training set contains 30,162 instances and 10 features (55 after one-hot encoding). The test set size is 15,060. |
| Hardware Specification | Yes | Experiments were run on a machine with 96 cores equipped with Intel(R) Xeon(R) Gold 6342 CPU @ 2.80GHz and two NVIDIA RTX A6000, OS Ubuntu 20.04.4, programming language Python 3.8.12. |
| Software Dependencies | Yes | OS Ubuntu 20.04.4, programming language Python 3.8.12. |
| Experiment Setup | Yes | For SELNET and SAT approach, we set 300 epochs in training, Stochastic Gradient Descent as an optimizer, a learning rate of .1 decreased by a factor .5 every 25 epochs, as in the original papers. All the parameters of base classifiers are left as the default ones. Regarding SCROSS, we fix K = 5 unless otherwise specified. |