Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in [1].
Exact Optimization of Conformal Predictors via Incremental and Decremental Learning
Authors: Giovanni Cherubin, Konstantinos Chatzikokolakis, Martin Jaggi
ICML 2021 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We evaluate our findings empirically, and discuss when methods are suitable for CP optimization. and We empirically compare our techniques with i) original implementations of full CP, and ii) the most computationally efficient CP modification, ICP. |
| Researcher Affiliation | Academia | 1Alan Turing Institute, London, UK 2University of Athens 3EPFL. |
| Pseudocode | Yes | Algorithm 1 CP: computing a p-value for (x, ˆy) |
| Open Source Code | Yes | Code to reproduce the experiments: https://github. com/gchers/exact-cp-optimization. |
| Open Datasets | Yes | (In Appendix G, we further compare CP and ICP on the MNIST dataset.) |
| Dataset Splits | No | The paper uses generated data for its main experiments and does not specify explicit train/validation/test splits, percentages, or cross-validation methodology for its evaluation. |
| Hardware Specification | No | The paper does not provide specific details about the hardware used for running the experiments. |
| Software Dependencies | No | The paper mentions using the 'scikit-learn library' but does not provide specific version numbers for any software dependencies. |
| Experiment Setup | Yes | For every training size n, chosen in the space [10, 105], we train the CP with a nonconformity measure, and use it to predict 100 test points. We set a timeout of 10 hours... We generate data for a binary classification problem with 30 features, by using the make classification() routine of the scikit-learn library. and We fix t/n = 0.5. |