CLUB: A Contrastive Log-ratio Upper Bound of Mutual Information
Authors: Pengyu Cheng, Weituo Hao, Shuyang Dai, Jiachang Liu, Zhe Gan, Lawrence Carin
ICML 2020 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Simulation studies on Gaussian distributions show the reliable estimation ability of CLUB. Real-world MI minimization experiments, including domain adaptation and information bottleneck, demonstrate the effectiveness of the proposed method. |
| Researcher Affiliation | Collaboration | 1Department of Electrical and Computer Engineering, Duke University, Durham, North Carolina, USA 2Microsoft, Redmond, Washington, USA. |
| Pseudocode | Yes | Algorithm 1 MI Minimization with v CLUB |
| Open Source Code | Yes | The code is at https://github.com/Linear95/CLUB. |
| Open Datasets | Yes | Following the same setup from Alemi et al. (2016), we apply the IB technique in the permutation-invariant MNIST classification. We compare performance on several DA benchmark datasets, including MNIST, MNIST-M, USPS, SVHN, CIFAR-10, and STL. |
| Dataset Splits | No | No explicit training/validation/test dataset splits were provided in the main text. The paper mentions 'Detailed description to the datasets and model setups is in the Supplementary Material,' but does not provide the information directly. |
| Hardware Specification | No | No specific hardware details (e.g., GPU/CPU models, memory amounts) used for running experiments were mentioned in the paper. |
| Software Dependencies | No | The paper mentions the use of neural networks and gradient-descent frameworks, but does not provide specific software dependencies with version numbers (e.g., Python, PyTorch/TensorFlow versions). |
| Experiment Setup | Yes | The approximation network has the same structure for all upper bounds, parameterized in a Gaussian family, qθ(y|x) = N(y|µ(x), σ2(x) I) with mean µ(x) and variance σ2(x) inferred by neural networks. On the top of hidden layer outputs, we add the ReLU activation function. The learning rate for all estimators is set to 5e-3. For both the Gaussian and Cubic setups, the number of hidden units of our CLUB estimator is set to 15. |