Multiclass Performance Metric Elicitation
Authors: Gaurush Hiranandani, Shant Boodaghians, Ruta Mehta, Oluwasanmi O. Koyejo
NeurIPS 2019 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | In this section, we empirically validate the results of theorems 1 and 2 and investigate sensitivity due to finite sample estimates. ... We consider two real-world datasets: (a) Sens IT (Acoustic) dataset [5] (78823 instances, 3 classes), and (b) Vehicle dataset [21] (846 instances, 4 classes). ... Some results are shown in Table 2 and Table 3. Results verify that we elicit the true metrics even for small ϵ = 0.01... |
| Researcher Affiliation | Academia | Gaurush Hiranandani Department of Computer Science University of Illinois at Urbana-Champaign gaurush2@illinois.edu Shant Boodaghians Department of Computer Science University of Illinois at Urbana-Champaign boodagh2@illinois.edu Ruta Mehta Department of Computer Science University of Illinois at Urbana-Champaign rutameht@illinois.edu Oluwasanmi Koyejo Department of Computer Science University of Illinois at Urbana-Champaign sanmi@illinois.edu |
| Pseudocode | Yes | The paper includes 'Algorithm 1: DLPM Elicitation' and 'Algorithm 2: LPM Elicitation', which are presented as structured pseudocode blocks. |
| Open Source Code | No | The paper does not contain any explicit statement about releasing source code or a link to a code repository for the described methodology. |
| Open Datasets | Yes | We consider two real-world datasets: (a) Sens IT (Acoustic) dataset [5] (78823 instances, 3 classes), and (b) Vehicle dataset [21] (846 instances, 4 classes). |
| Dataset Splits | No | For all the datasets, we standardize the features and split the dataset into two parts S1 and S2. On S1, we learn {ˆηi(x)}k i=1 using a regularized softmax regression model. We use S2 for making predictions and computing sample confusions. |
| Hardware Specification | No | The paper does not provide specific details on CPU models, GPU models, or other hardware specifications used for experiments. It only mentions 'Gaurush Hiranandani and Oluwasanmi Koyejo thank Microsoft Azure for providing computing credits', which implies cloud resources but no specific hardware models. |
| Software Dependencies | No | The paper mentions learning {ˆηi(x)}k i=1 using a regularized softmax regression model, but does not specify any software libraries or their version numbers. |
| Experiment Setup | Yes | For all the datasets, we standardize the features and split the dataset into two parts S1 and S2. On S1, we learn {ˆηi(x)}k i=1 using a regularized softmax regression model. We use S2 for making predictions and computing sample confusions. ... To verify elicitation, we first define a true metric ψ or φ . This specifies the query outputs of Algorithm 1 or Algorithm 2. Then we run the algorithms to check whether or not we recover the same metric. ... For the full Sens IT (Acoustic) dataset, we elicit all the metrics within ω = 0.12. We randomly selected 100 DLPMs i.e. a s. We then used Algorithm 1 with ϵ = 0.01 to recover the estimates ˆa s. |