Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in [1].
Towards Backwards-Compatible Data with Confounded Domain Adaptation
Authors: Calvin McCarter
TMLR 2024 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Finally, we demonstrate our approach on synthetic and real datasets. We compare Con Do to baseline methods on three synthetic, two hybrid, and three real data settings. When available, we evaluate against the latent ground-truth features by measuring the root-mean-squared error (r MSE) between these and the adapted features. We also compare the different methods on unsupervised domain adaptation (UDA), measuring the performance of source-trained classifiers on adapted target test data, to evaluate our ability to adapt data for use by non-retrainable downstream models. |
| Researcher Affiliation | Academia | Calvin Mc Carter EMAIL. The paper does not explicitly state the author's institutional affiliation, only a personal email address. However, given it is published in "Transactions on Machine Learning Research", an academic journal, it is classified as academic work. |
| Pseudocode | No | The paper describes algorithms and methods but does not present them in structured pseudocode or algorithm blocks. The methods are explained through mathematical formulations and textual descriptions. |
| Open Source Code | Yes | Our software, with a Scikit-learn (Pedregosa et al., 2011) compatible API and with experimental scripts, is available at https://github.com/calvinmccarter/condo-adapter. |
| Open Datasets | Yes | We next evaluated approaches on the ANSUR II (Gordon et al., 2014) dataset... Here, we apply Con Do to an unsupervised domain adaptation setting derived from the California housing dataset (Pace & Barry, 1997)... We apply domain adaptation to a SNAREseq single-cell multi-omics dataset (Demetci et al., 2022)... We analyze performance on the bladderbatch gene expression dataset commonly used to benchmark batch correction methods (Dyrskjøt et al., 2004; Leek, 2016). |
| Dataset Splits | Yes | We split the data into source and target domains based on the first feature, Median Income, defining the source domain as being the housing districts with income less than or equal to the median... repeated over 10 random simulations, each with 500 source training samples and 500 target test samples in each simulation. In each simulation, we have a large source domain training set of 500 source cells. We also simulate having a small set of C cells for which we have both source and target domain data available to use for adaptation; we simulate C {5, 10, 20, 50, 100}. |
| Hardware Specification | No | The paper discusses the training and evaluation of models but does not specify any particular hardware (e.g., GPU/CPU models, processor types, or memory amounts) used for these experiments. |
| Software Dependencies | No | We optimize the Con Do-KLD objective with the Py Torch-minimize (Feinman, 2021) implementation of the trust-region Newton conjugate gradient method (Lin & Jorge, 1999). We optimize the MMD and Con Do-MMD objectives with Adam W (Loshchilov & Hutter, 2017)... as implemented in MICE-Forest (Wilson et al., 2022) with Light GBM (Ke et al., 2017). The paper mentions various software components and libraries but does not provide specific version numbers for them. |
| Experiment Setup | Yes | We optimize the MMD and Con Do-MMD objectives with Adam W (Loshchilov & Hutter, 2017), with weight decay = 10 4, β1 = 0.9, and β2 = 0.999. We set KX = 20 (for Con Do-KLD and Con Do-MMD) and KZ = 8 (for Con Do-MMD); for MMD, each step also uses 8 evaluations of the MMD loss with 20 samples. We run for 5 epochs (with an early stopping patience of 3 epochs) with learning rate = 10 3, unless otherwise indicated in the Appendix; for both MMD and Con Do-MMD, an epoch is defined as min(NS, NT )/8 steps. |