Overcoming Saturation in Density Ratio Estimation by Iterated Regularization
Authors: Lukas Gruber, Markus Holzleitner, Johannes Lehner, Sepp Hochreiter, Werner Zellinger
ICML 2024 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We investigate the performance of iterated density ratio estimation regarding three aspects below. We include additional experiments conducted during the rebuttal period that helped to improve our work. More precisely, we compare to a SOTA domain adaptation method (Dinu et al., 2023), combine our approach with telescoping from Rhodes et al. (2020), and extend our approach to Deep Learning methods. Please find further details in Appendix E. Sample convergence in highly regular problems. To investigate the sample convergence of iterated approaches compared to their non-iterated regularized versions, i.e., Theorem 1), we follow the study of Beugnot et al. (2021). ... Accuracy improvement for known density ratios. To study the accuracy of the iteratively regularized estimations of density ratios, we follow investigations of Kanamori et al. (2012b) ... Importance weighted ensembling in deep domain adaptation. To test the effect of iterated regularization in a real real world setting we rely on large-scale (over 9000 trained neural networks) state-of-the-art experiments for re-solving parameter choice issues in unsupervised domain adaptation. |
| Researcher Affiliation | Collaboration | 1ELLIS Unit Linz and LIT AI Lab, Institute for Machine Learning, Johannes Kepler University Linz, Austria 2Ma LGa Center, Department of Mathematics, University of Genoa 3NXAI Gmb H, Linz, Austria 4Johann Radon Institute for Computational and Applied Mathematics, Austrian Academy of Sciences. |
| Pseudocode | No | The paper describes algorithmic realization in prose but does not contain a structured pseudocode or algorithm block. |
| Open Source Code | Yes | A reference implementation of the methods presented in this paper is available at: https://github.com/lugruber/dre_iter_reg. |
| Open Datasets | Yes | To illustrate Theorem 1, we adapt an example of Beugnot et al. (2021, Section 4)... Following Kanamori et al. (2012b), we generate ten different datasets... utilize the breast cancer dataset (Street et al., 1993)... The Amazon reviews dataset (Blitzer et al., 2006)... The Domain Net-2019 dataset (Peng et al., 2019)... The Heterogeneity Human Activity Recognition (Stisen et al., 2015) dataset... |
| Dataset Splits | Yes | The selection and evaluation of the compared density ratio estimation method is done in a typical train/val/test split approach with split ratios 64/16/20 respectively. ... For training and selecting the density ratio estimation methods within this pipeline we perform an additional train/val split of 80/20 on the datasets that are used for training the domain adaption methods. |
| Hardware Specification | No | No specific hardware details (e.g., GPU/CPU models, memory amounts, or detailed computer specifications) used for running experiments were mentioned. |
| Software Dependencies | Yes | We rely on the CG method implemented in Python Scipy (Virtanen et al., 2020)... |
| Experiment Setup | Yes | The regularization (hyper) parameter λ is selected from {10 6, 10 5, . . . , 104} and for each experiment 10 replicates are carried out. Accordingly, the number of iteration steps is selected from {1, 2, . . . , 10} based on the respective loss metric. We follow (Kanamori et al., 2012a) in using the Gaussian kernel with kernel width set according to the median heuristic (Sch olkopf & Smola, 2002) for all compared density ratio estimation methods. |