Scalable Optimization of Multivariate Performance Measures in Multi-instance Multi-label Learning
Authors: Apoorv Aggarwal, Sandip Ghoshal, Ankith Shetty, Suhit Sinha, Ganesh Ramakrishnan, Purushottam Kar, Prateek Jain
AAAI 2017 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We present a novel method for optimizing multivariate performance measures in the MIML setting. Our approach MIMLperf uses a novel plug-in technique and offers a seamless way to optimize a vast variety of performance measures such as macro and micro-F measure, average precision, which are performance measures of choice in multi-label learning domains. MIMLperf offers two key benefits over the state of the art. Firstly, across a diverse range of benchmark tasks, ranging from relation extraction to text categorization and scene classification, MIMLperf offers superior performance as compared to state of the art methods designed specifically for these tasks. 5 Experiments We present detailed comparisons of our approach with the state of the art on three benchmark MIML/RE datasets. |
| Researcher Affiliation | Collaboration | Indian Institute of Technology Bombay, Indian Institute of Technology Kanpur, Microsoft Research |
| Pseudocode | Yes | Algorithm 1 MIMLperf: Training Routine, Algorithm 2 MIMLperf: Testing Routine |
| Open Source Code | No | The paper does not explicitly state that source code for the described methodology is being released, nor does it provide a direct link to a code repository. Footnotes link to supplementary material for details and a spreadsheet of results, but not code. |
| Open Datasets | Yes | Riedel Distant Supervision Dataset: For the distantly supervised relation extraction problem, we use the benchmark dataset created by (Riedel, Yao, and Mc Callum 2010). MIML Scene Classification Dataset (Scene): The Scene data set contains 2000 scene images collected from the COREL image collection and the Internet... MIML Text Classification Dataset (Reuters): The text data is derived from the widely studied Reuters-21578 collection using seven most frequent classes. |
| Dataset Splits | No | For the Scene dataset, the paper states: 'We divided the data into two parts consisting of 1600 data points for training and the remaining 400 points for testing.' For the Reuters dataset, it states: 'Again, we follow MIMLSVM in the way we partition this dataset into training and testing splits.' While training and testing splits are mentioned, a separate validation split is not explicitly provided or referenced for reproducibility. |
| Hardware Specification | No | The paper does not explicitly describe the hardware used for running its experiments, such as specific CPU or GPU models, or cloud infrastructure specifications. |
| Software Dependencies | No | The paper mentions methods like logistic regression and structural SVM, but it does not specify any software dependencies with version numbers (e.g., Python 3.8, PyTorch 1.9, or specific library versions) that would be needed to replicate the experiments. |
| Experiment Setup | Yes | We used a default value κ = 1 as the prevalence parameter for training MIMLperf. For the initialization step, if a bag has a label j, the method assigns a random κ fraction of instances in that bag to label j. κ is an expression rate parameter that is only used for initialization. Algorithm 1 MIMLperf: Training Routine. |