Fine-Tuned Language Models Generate Stable Inorganic Materials as Text
Authors: Nate Gruver, Anuroop Sriram, Andrea Madotto, Andrew Gordon Wilson, C. Lawrence Zitnick, Zachary Ward Ulissi
ICLR 2024 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We evaluate our method with Materials Project data (Jain et al., 2013), comparing against an invariant diffusion model and a sequence model trained from scratch. |
| Researcher Affiliation | Collaboration | 1NYU 2Meta FAIR |
| Pseudocode | Yes | Listing 1: Self contained code to construct the template method table which can be used to proposed mutations for local optimization around an existing material. Listing 2: Self contained code implementing a template method with uniform sampling. |
| Open Source Code | Yes | 1https://github.com/facebookresearch/crystal-llm |
| Open Datasets | Yes | We evaluate our method with Materials Project data (Jain et al., 2013) and For consistency with prior work (Xie et al., 2021; Flam-Shepherd et al., 2023) we used MP-20 (Jain et al., 2013), a dataset of 45231 materials, when training for unconditional generation. |
| Dataset Splits | Yes | The original validation and test splits are left unchanged and all test/validation points are removed from the new training set. |
| Hardware Specification | Yes | We run LLa MA-2 models with the largest feasible batch size on one A100 GPU (Appendix B.7). and We ran experiments primarily on A100 GPUs, while the publicly available code for CDVAE cannot be run on an A100 and reports results on a RTX2080 Ti. and Considering AWS as the deployment environment, we can build on a recent benchmark on a cloud instance with 8 A100 GPUs (ml.p4d.12xlarge) (Schmid, 2023) |
| Software Dependencies | Yes | All of our experiments were conducted with LLa MA-2 models (7B 13B, and 70B) (Touvron et al., 2023a;b) through the Transformers library (Wolf et al., 2020) and Py Torch (Paszke et al., 2019). and DFT: We run a relaxation using the Density Functional Theory code VASP (Hafner, 2008) with INCAR settings chosen by Pymatgen (Ong et al., 2013). and VASP relaxations were run using the GPU-accelerated VASP6 code. |
| Experiment Setup | Yes | We provide the full hyperparameters and training details in Appendix A.4. and LLa MA-2 7B: Batch size of 256 for 65 epochs with a cosine annealed learning rate of 0.0005. Lo RA rank 8 and alpha 32. LLa MA-2 13B: Batch size of 256 for 44 epochs with a cosine annealed learning rate of 0.0005. Lo RA rank 8 and alpha 32. LLa MA-2 70B: Batch size of 32 for 21 epochs with a cosine annealed learning rate of 0.0005. Lo RA rank 8 and alpha 32. |