Bridging Model-Based Optimization and Generative Modeling via Conservative Fine-Tuning of Diffusion Models
Authors: Masatoshi Uehara, Yulai Zhao, Ehsan Hajiramezanali, Gabriele Scalia, Gokcen Eraslan, Avantika Lal, Sergey Levine, Tommaso Biancalani
NeurIPS 2024 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Through empirical and theoretical analysis, we demonstrate the capability of our approach to outperform the best designs in offline data, leveraging the extrapolation capabilities of reward models while avoiding the generation of invalid designs through pretrained diffusion models. We perform experiments to evaluate (a) the effectiveness of conservative methods for fine-tuning diffusion models and (b) the comparison of our approach between existing methods for MBO with diffusion models (Krishnamoorthy et al., 2023; Yuan et al., 2023). |
| Researcher Affiliation | Collaboration | Masatoshi Uehara 1 Yulai Zhao 2 Ehsan Hajiramezanali 1 Gabriele Scalia 1 Gokcen Eraslan 1 Avantika Lal 1 Sergey Levine3 Tommaso Biancalani 1 1Genentech 2Princeton University 3UC Berkeley |
| Pseudocode | Yes | Algorithm 1 BRAID (dou Bly conse Rv Ative f Ine-tuning Diffusion models) |
| Open Source Code | Yes | The main code is available at https://github. com/masa-ue/RLfinetuning_Diffusion_Bioseq. |
| Open Datasets | Yes | We examine two publicly available large datasets consisting of enhancers (n 700k) (Gosai et al., 2023) and UTRs (n 300k) (Sample et al., 2019)... We use the AVA dataset (Murray et al., 2012) as our offline data... |
| Dataset Splits | No | The paper describes using a limited number of online interactions for hyperparameter selection and divides the dataset for oracle construction and testing, but it does not provide specific percentages or counts for standard training/validation/test dataset splits for the main model. |
| Hardware Specification | Yes | In all experiments, we use A100 GPUs. In all image experiments, we use four A100 GPUs for fine-tuning Stable Diffusion v1.5 (Rombach et al., 2022). |
| Software Dependencies | No | The paper mentions optimizers (Adam, AdamW) and models (Stable Diffusion v1.5, DDIM scheduler) but does not provide specific version numbers for key software dependencies like programming languages or deep learning frameworks (e.g., Python, PyTorch, TensorFlow, CUDA). |
| Experiment Setup | Yes | Table 2: Important hyperparameters for fine-tuning. For all methods, we use Adam as an optimizer. Method Type Value Batch size 128 KL parameter α 0.001 LCB parameter (bonus) c 0.1 (UTRs), 0.1 (Enhancers) Number of bootstrap heads 3 Sampling to neural SDE Euler Maruyama Step size (fine-tuning) 50 Guidance Guidance level 10 Guidance target Top 5%. Table 4: Important hyperparameters for fine-tuning Aesthetic Scores. Method Parameters Values Guidance weight 7.5 DDIM Steps 50 Batch size 128 KL parameter α 1 LCB bonus parameter C 0.001 Number of bootstrap heads 4 Optimization Optimizer Adam W Learning rate 0.001 (ϵ1, ϵ2) (0.9, 0.999) Weight decay 0.1 Clip grad norm 5 Truncated back-propagation step K K Uniform(0, 50). |