Learning Invariant Representations using Inverse Contrastive Loss
Authors: Aditya Kumar Akash, Vishnu Suresh Lokhande, Sathya N. Ravi, Vikas Singh6582-6591
AAAI 2021 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Our experimental results indicate that models obtained by optimizing ICL achieve significantly better invariance to the extraneous variable for a fixed desired level of accuracy. In a variety of experimental settings, we show applicability of ICL for learning invariant representations for both continuous and discrete extraneous variables. |
| Researcher Affiliation | Academia | Aditya Kumar Akash1, Vishnu Suresh Lokhande1, Sathya N. Ravi2, Vikas Singh1 1 University of Wisconsin-Madison, 2 University of Illinois at Chicago aakash@wisc.edu, lokhande@cs.wisc.edu, sathya@uic.edu, vsingh@biostat.wisc.edu |
| Pseudocode | No | The paper includes mathematical derivations and proofs, but no explicit pseudocode or algorithm blocks are provided. |
| Open Source Code | Yes | The project page with code is available at https://github.com/adityakumarakash/ICL |
| Open Datasets | Yes | Learning style information in MNIST Dataset. ... We use the Adult and German datasets (Dua and Graff 2017) for this task. ... Predicting disease status while controlling for scanner confounds (ADNI dataset (adni.loni.usc.edu)). |
| Dataset Splits | Yes | The hyperparameter selection is done on a validation split such that best adversarial invariance is achieved for task accuracy within 5% of unregularized model for supervised tasks and reconstruction MSE within 5 points of unregularized model for unsupervised tasks. ... The ADNI dataset is very small and hence for this dataset we use five fold random training validation splits to report the mean and standard deviation. |
| Hardware Specification | No | The paper does not provide specific details regarding the hardware used for running experiments, such as GPU models, CPU types, or memory specifications. |
| Software Dependencies | No | The paper mentions 'Adam optimizer' but does not specify version numbers for any software dependencies or libraries used in the experiments. |
| Experiment Setup | Yes | We use Adam optimizer for model training. ... The hyperparameter selection is done on a separate validation split such that on this set the model achieves the best adversarial invariance while the task accuracy remains within 5% of the unregularized model for supervised tasks and within 5 points of the unregularized model for unsupervised tasks. For the baselines, we grid search the best regularization weight in powers of ten and select the one with best invariance on validation set. For some of the experiments, we found it useful to initialize the regularization weight to a smaller value (0.01 times the regularizer weight) and multiplicatively update it (with factor 1.5) every epoch till it reaches the best found regularization weight. The same update rule is used for all the baselines. |