Parallel-mentoring for Offline Model-based Optimization
Authors: Can (Sam) Chen, Christopher Beckham, Zixuan Liu, Xue (Steve) Liu, Chris Pal
NeurIPS 2023 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Experiments validate the effectiveness of our method. Our code is available here. We conduct extensive experiments on design-bench to investigate the effectiveness and robustness of the proposed method. In Section 4.4, we benchmark our approach against several well-established baselines. In Section 4.5, we verify the effectiveness of two modules: voting-based pairwise supervision and adaptive soft-labeling, as well as other contributing factors. In Section 4.6, we investigate the sensitivity of our method to hyperparameter changes. |
| Researcher Affiliation | Academia | Can (Sam) Chen1,2 , Christopher Beckham2,3, Zixuan Liu5, Xue Liu1,2, Christopher Pal2,3,4 1Mc Gill University, 2MILA Quebec AI Institute, 3Polytechnique Montreal, 4Canada CIFAR AI Chair, 5University of Washington |
| Pseudocode | Yes | Algorithm 1 Tri-mentoring for Offline Model-based Optimization |
| Open Source Code | Yes | Our code is available here. |
| Open Datasets | Yes | We adopt the design-bench which comprises both continuous and discrete tasks. Dataset. We conduct experiments on four continuous tasks: (1) Superconductor (Super C) [2]: discover an 86-D superconductor to maximize critical temperature with 17010 designs. (2) Ant Morphology (Ant) [20]: identify a 60-D ant morphology to crawl quickly with 10004 designs. (3) D Kitty Morphology (D Kitty) [21]: determine a 56-D D Kitty morphology to crawl quickly with 10004 designs. (4) Hopper Controller (Hopper) [1]: find a neural network policy with 5126 weights to maximize return with 3200 designs. Besides, we perform experiments on four discrete tasks: (1) TF Bind 8 (TFB8) [5]: design a length 8 DNA sequence to maximize binding activity score with 32896 designs. (2) TF Bind 10 (TFB10) [5]: find a length 10 DNA sequence to maximize binding activity score with 50000 designs. (3) NAS [1]: find a 64-D NN with 5 categories per dimension to maximize the performance on CIFAR10 with 1771 designs. |
| Dataset Splits | No | The paper mentions training on a "static dataset D" and using a bi-level optimization where the inner level fine-tune proxies and the outer level refines soft-labels by minimizing loss on the static dataset. However, it does not explicitly provide percentages, sample counts, or predefined citations for training, validation, and test splits of the initial static dataset D, which would be needed for direct reproduction of dataset partitioning. |
| Hardware Specification | Yes | All experiments are performed on a single V100 GPU. |
| Software Dependencies | No | The paper mentions using a "three-layer MLP network with the Re LU function as the activation" and an "Adam optimizer." It also references "higher, a library for higher-order optimization [19]". However, it does not provide specific version numbers for any of these software components (e.g., Adam optimizer version, ReLU version, or the version of the 'higher' library), which are necessary for reproducible software dependencies. |
| Experiment Setup | Yes | We adopt a three-layer MLP network with the Re LU function as the activation. We train the MLP model on the static dataset with a 1 10 3 learning rate and an Adam optimizer. The fine-tuning learning rate γ is set as 1 10 3 and the soft-labeling learning rate λ is set as 1 10 1. The standard deviation δ is set as 1 10 1 and the number of the samples K is set as 10. |