Multimodal Molecular Pretraining via Modality Blending
Authors: Qiying Yu, Yudi Zhang, Yuyan Ni, Shikun Feng, Yanyan Lan, Hao Zhou, Jingjing Liu
ICLR 2024 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Extensive experiments show that MOLEBLEND achieves state-of-the-art performance across major 2D/3D molecular benchmarks. We further provide theoretical insights from the perspective of mutual-information maximization, demonstrating that our method unifies contrastive, generative (cross-modality prediction) and mask-then-predict (single-modality prediction) objectives into one single cohesive framework. |
| Researcher Affiliation | Collaboration | Qiying Yu1 , Yudi Zhang2 , Yuyan Ni3, Shikun Feng1, Yanyan Lan1,4, Hao Zhou1,5, , Jingjing Liu1 1 Institute for AI Industry Research, Tsinghua University 2 Harbin Institute of Technology 3 Academy of Mathematics and Systems Science, Chinese Academy of Sciences 4 Beijing Academy of Artificial Intelligence 5 Shanghai Artificial Intelligence Laboratory |
| Pseudocode | No | The paper does not contain structured pseudocode or algorithm blocks. |
| Open Source Code | No | The paper does not provide any statement or link indicating that the source code for the methodology is openly available. |
| Open Datasets | Yes | For pretraining, we use PCQM4Mv2 dataset from the OGB Large-Scale Challenge (Hu et al., 2021), which includes 3.37 million molecules with both 2D graphs and 3D geometric structures. To evaluate the versatility of MOLEBLEND, we carry out extensive experiments on 24 molecular tasks with different data formats across three representative benchmarks: Molecule Net (Wu et al., 2017) (2D, 11 tasks), QM9 quantum properties (Ramakrishnan et al., 2014) (3D, 12 tasks), and PCQM4Mv2 humo-lumo gap (2D). Further details about these datasets can be found in the Appendix C.1. |
| Dataset Splits | Yes | QM9 is a quantum chemistry benchmark with 134K small organic molecules. It contains 12 tasks, covering the energetic, electronic and thermodynamic properties of molecules. Following (Thölke & Fabritiis, 2022), we randomly split 10,000 and 10,831 molecules as validation and test set, and use the remaining molecules for finetuning. |
| Hardware Specification | No | The paper does not provide specific details about the hardware (e.g., GPU models, CPU types) used to run the experiments. It mentions using a 'Transformer' as the backbone model, but no hardware specifics. |
| Software Dependencies | No | The paper mentions 'Adam W optimizer' but does not specify version numbers for any software or libraries used in the implementation. |
| Experiment Setup | Yes | For pretraining, we use Adam W optimizer and set (β1, β2) to (0.9, 0.999) and peak learning rate to 1e-5. Batch size is 4096. We pretrain the model for 1 million steps with initial 100k steps as warm-up, after which learning rate decreases to zero with cosine scheduler. The blending ratio p is 2:2:6, and the ablations on p can be found in Appedix A.3. Table 8: Hyperparameters setup for pretraining. Table 9: Search space for Molecule Net tasks. Table 10: Hyperparameters for QM9 finetuning. |