Synthesize, Partition, then Adapt: Eliciting Diverse Samples from Foundation Models
Authors: Yeming Wen, Swarat Chaudhuri
NeurIPS 2024 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | To demonstrate the effectiveness of our approach, we conduct experiments on a range of tasks in both the code generation and natural language understanding domains. We evaluate our method on the Human Eval (Chen et al., 2021) and MBPP (Austin et al., 2021) datasets for code generation, as well as several natural language understanding tasks. |
| Researcher Affiliation | Academia | Yeming Wen & Swarat Chaudhuri Department of Computer Science The University of Texas at Austin |
| Pseudocode | No | The paper includes Figure 3 as an illustration of the SPA framework, but it is a diagram, not structured pseudocode or an algorithm block. |
| Open Source Code | Yes | The code is included in the supplementary material. |
| Open Datasets | Yes | For the synthetic dataset, we utilize the OSS-Instruct dataset (Wei et al., 2023)... For the synthetic dataset, we use Platypus (Lee et al., 2023)... |
| Dataset Splits | No | The paper describes partitioning the synthetic datasets into K groups for training different model adaptations, but it does not specify explicit training/validation/test splits for the overall experimental evaluation of the models. |
| Hardware Specification | Yes | For the code generation experiments, we use a machine with 3 A100 40GB GPUs and train each partition for 400 steps... On the OSS-Instruct dataset, this takes roughly 5 hours using a single A100 80GB GPU. |
| Software Dependencies | No | The paper mentions software components like 'LORA', 'BM25', 'Python' and 'torch-influence' but does not specify their version numbers. |
| Experiment Setup | Yes | For the code generation experiments, ... The base model is Code LLa MA-7B-Python, and we use bf16 precision to accelerate training. The per-device train batch size is set to 1, with a gradient accumulation step of 20. We use a learning rate of 2e-4 with a cosine learning rate scheduler and 20 warmup steps. For the LORA hyperparameters, we use a rank (r) of 16, an alpha of 16. |