Improving Few-Shot Text-to-SQL with Meta Self-Training via Column Specificity
Authors: Xinnan Guo, Yongrui Chen, Guilin Qi, Tianxing Wu, Hao Xu
IJCAI 2022 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Extensive experimental results on both open-domain and domain-specific benchmarks reveal that our MST-SQL1 has significant advantages in few-shot scenarios, and is also competitive in standard supervised settings. |
| Researcher Affiliation | Collaboration | Xinnan Guo1 , Yongrui Chen1 , Guilin Qi1 , Tianxing Wu1 and Hao Xu2 1School of Computer Science and Engineering, Southeast University, Nanjing, China 2Zhejiang Lab, Zhejiang, China |
| Pseudocode | Yes | Algorithm 1 details our meta self-training framework F, used to find the optimal parameters θ by mining the potential knowledge of unlabeled data in the following two stages. Our CSML is detailed in Algorithm 2. |
| Open Source Code | Yes | Code available at https://github.com/ygxw0909/MST-SQL Contact Author |
| Open Datasets | Yes | We evaluated our MST-SQL on the two benchmarks: 1) Wiki SQL [Zhong et al., 2017] is currently the largest open-domain single-table text-to-SQL dataset, which contains more than 20k tables collected from Wikipedia and 56,355 / 8,421 / 15,878 NLQ-SQL pairs for training / development / test. 2) ESQL [Chen et al., 2021] is a Chinese domain-specific singletable text-to-SQL dataset containing 17 tables and 10,000 / 1,000 / 2,000 NLQ-SQL pairs for training / development / test. |
| Dataset Splits | Yes | We evaluated our MST-SQL on the two benchmarks: 1) Wiki SQL [Zhong et al., 2017] is currently the largest open-domain single-table text-to-SQL dataset, which contains more than 20k tables collected from Wikipedia and 56,355 / 8,421 / 15,878 NLQ-SQL pairs for training / development / test. 2) ESQL [Chen et al., 2021] is a Chinese domain-specific singletable text-to-SQL dataset containing 17 tables and 10,000 / 1,000 / 2,000 NLQ-SQL pairs for training / development / test. |
| Hardware Specification | Yes | Our method ran on Tesla V100 Super GPUs. |
| Software Dependencies | No | The paper mentions using "Ro BERTa as the encoder by default" but does not specify any software dependencies with version numbers (e.g., Python, PyTorch, or specific library versions). |
| Experiment Setup | Yes | The hyper-parameters were set as follow: (1) The number of table content was set to 5. (2) The learning rate α, β, γ were set to 3e-5, 1e-5, 3e-5. (3) The weight of the loss calculation in CSML η was set to 0.5. (4) The threshold for confidence scores ζ was set to 0.1. (5) N was set to 4, and K was set to 15 / 5 for support / query set. (6) ML-task number n, sampling ratio σ, and warm-boot threshold λ were set to 80, 0.2, 80 in standard supervised setting, and the setting in few-shot tests will be discussed later. |