A General Online Algorithm for Optimizing Complex Performance Metrics
Authors: Wojciech Kotlowski, Marek Wydmuch, Erik Schultheis, Rohit Babbar, Krzysztof Dembczynski
ICML 2024 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We also verify the efficiency of the proposed algorithm in empirical studies, where we evaluate it on a wide range of multi-label benchmarks and performance measures. |
| Researcher Affiliation | Collaboration | 1Poznan University of Technology, Poznan, Poland 2Aalto University, Helsinki, Finland 3University of Bath, Bath, UK 4Yahoo Research, New York, United States. |
| Pseudocode | Yes | Algorithm 1 Online Measure Maximization Algorithm; Algorithm 2 OMMA(bη) |
| Open Source Code | Yes | Code to reproduce the experiments: https://github.com/mwydmuch/x COLUMNs |
| Open Datasets | Yes | For multi-class experiments, we use News20 (Lang, 1995), Ledgar-Lex Glue (Chalkidis et al., 2022) with tf-idf features, Caltech-256 (Griffin et al., 2007) with features obtained using VGG16 (Simonyan & Zisserman, 2014) trained on Image Net, and for multi-label experiments, we use You Tube, Flickr (Tang & Liu, 2009) with Deep Walk features (Perozzi et al., 2014), Eurlex-Lex Glue (Chalkidis et al., 2021), Mediamill (Snoek et al., 2006), RCV1X (Lewis et al., 2004), and Amazon Cat (Mc Auley & Leskovec, 2013; Bhatia et al., 2016) with tf-idf features. |
| Dataset Splits | Yes | For benchmarks without default train and test sets, we split them randomly in proportion 70/30. |
| Hardware Specification | No | The paper states: 'All the experiments were conducted on a workstation with 64 GB of RAM.' This is a general description and does not provide specific details such as CPU/GPU models, processor types, or exact memory amounts (e.g., type of RAM). |
| Software Dependencies | Yes | In multi-label experiments with fixed CPE, we use LIBLINEAR model with L2-regularized logistic loss (Fan et al., 2008). In multi-class experiments with both fixed and online CPE, we use Py Torch (Paszke et al., 2019) linear model with L2-regularized binary cross entropy or cross entropy loss, optimized using ADAM (Kingma & Ba, 2015). |
| Experiment Setup | Yes | As discussed in Remark 4.4, we add both small constant = 1e-9 to the denominators in all metrics, as well as the regularization value {0, 1e-6, 1e-3, 0.1, 1} to the entries of the confusion matrix used in all online algorithms (Greedy, OFO, Online-FW, and OMMA). |