Effective Structured Prompting by Meta-Learning and Representative Verbalizer
Authors: Weisen Jiang, Yu Zhang, James Kwok
ICML 2023 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Experimental results demonstrate that Meta Prompter performs better than the recent state-of-the-arts and Rep Verb outperforms existing soft verbalizers. |
| Researcher Affiliation | Academia | 1Department of Computer Science and Engineering, Southern University of Science and Technology 2Department of Computer Science and Engineering, Hong Kong University of Science and Technology 3Peng Cheng Laboratory. |
| Pseudocode | Yes | Algorithm 1 Representative Verbalizer (Rep Verb).Algorithm 2 Meta Prompter. |
| Open Source Code | No | The paper does not provide explicit statements or links indicating that the source code for the described methodology is open-source or publicly available. |
| Open Datasets | Yes | Following Chen et al. (2022), we perform few-shot classification on six popularly used data sets: (i) 20News (Lang, 1995), which contains informal discourses from news discussion forums of 20 topics; (ii) Amazon (He & Mc Auley, 2016), which consists of customer reviews from 24 products... (iii) Huff Post (Misra, 2022)... (iv) Reuters (Lewis, 1997)... (v) HWU64 (Liu et al., 2019)... (vi) Liu54 (Liu et al., 2019). |
| Dataset Splits | Yes | We use the meta-training/meta-validation/meta-testing splits provided in Chen et al. (2022).To prevent overfitting, we evaluate the meta-validation performance every 50 iteration and choose the checkpoint with the best meta-validation performance for meta-testing. |
| Hardware Specification | Yes | Experiments are run on a DGX station with 8 V100 32GB GPUs. |
| Software Dependencies | No | The paper mentions using 'bert-base-uncased' from Hugging Face and the 'Adam optimizer', but does not specify software versions for programming languages, libraries, or frameworks (e.g., Python version, PyTorch/TensorFlow version). |
| Experiment Setup | Yes | For Meta Prompter, hyperparameters K and Lp are chosen from {1, 2, 4, 8, 16, 32, 64} using the meta-validation set. For the base learner, α = 0.1, and J = 5 (resp. 15) at metatraining (resp. meta-validation or meta-testing). We train the prompt pool for T = 3, 000 iterations using the Adam optimizer (Kingma & Ba, 2015) with a learning rate of 0.001. |