Few-Shot Conformal Prediction with Auxiliary Tasks
Authors: Adam Fisch, Tal Schuster, Tommi Jaakkola, Dr.Regina Barzilay
ICML 2021 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We demonstrate the effectiveness of this approach across a number of few-shot classification and regression tasks in natural language processing, computer vision, and computational chemistry for drug discovery. We empirically validate our approach on image classification, relation classification for textual entities, and chemical property prediction for drug discovery. |
| Researcher Affiliation | Academia | Adam Fisch 1 Tal Schuster 1 Tommi Jaakkola 1 Regina Barzilay 1 1Computer Science and Artificial Intelligence Laboratory, Massachusetts Institute of Technology, Cambridge, MA, USA. |
| Pseudocode | Yes | Pseudo-code for our meta CP procedure is given in Algorithm 1. Algorithm 1 Meta conformal prediction with auxiliary tasks. |
| Open Source Code | Yes | Our code is publicly available.1 1https://github.com/ajfisch/few-shot-cp. |
| Open Datasets | Yes | We use the mini Image Net dataset (Vinyals et al., 2016), a downsampled version of a subset of classes from Image Net (Deng et al., 2009). We use the Few Rel 1.0 dataset (Han et al., 2018), which consists of 100 relations derived from 70k Wikipedia sentences. We use the Ch EMBL dataset (Mayr et al., 2018), and regress the p Ch EMBL value (a normalized log-activity metric) for individual molecule-property pairs. |
| Dataset Splits | Yes | mini Image Net contains 100 classes that are divided into training, validation, and test class splits. Like mini Image Net, the relation types are divided into training, validation, and test splits. We select a subset of 296 assays from Ch EMBL, and divide them into training (208), validation (44), and test (44) splits. |
| Hardware Specification | No | No specific hardware details (e.g., GPU/CPU models, memory specifications) used for running experiments were mentioned in the paper. |
| Software Dependencies | No | The paper mentions several components like 'CNN encoder', 'GloVe embeddings', 'ridge regressor', and 'Message Passing Network molecular encoder', and their associated papers, but it does not specify software dependencies with version numbers (e.g., PyTorch 1.9, TensorFlow 2.x). |
| Experiment Setup | No | The paper provides some high-level experimental settings such as 'We use K = 16 and N = 10 in our experiments' and 'We report marginalized results over 5000 random trials'. However, it does not include specific hyperparameters (e.g., learning rates, batch sizes, optimizer details) for the training of the neural network components. |