Frugal LMs Trained to Invoke Symbolic Solvers Achieve Parameter-Efficient Arithmetic Reasoning
Authors: Subhabrata Dutta, Ishan Pandey, Joykirat Singh, Sunny Manchanda, Soumen Chakrabarti, Tanmoy Chakraborty
AAAI 2024 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | SYRELM shows massive improvements (e.g., +30.65 absolute point improvement in accuracy on the SVAMP dataset using GPT-J 6B model) over base LMs, while keeping our testbed easy to diagnose, interpret and within reach of most researchers. |
| Researcher Affiliation | Collaboration | 1IIT Delhi, India 2IIIT Delhi, India 3DYSL-AI, India 4IIT Bombay, India |
| Pseudocode | Yes | Figure 2: Example of parsable pseudocode generation from LMs. |
| Open Source Code | Yes | We release the code and data for reproducibility 2. 2https://github.com/joykirat18/SYRELM |
| Open Datasets | Yes | We built upon three existing mathword problem datasets: ASDiv (Miao, Liang, and Su 2020b), MAWPS (Miao, Liang, and Su 2020b), and Math23k (Shen et al. 2021). |
| Dataset Splits | No | The paper mentions 'training data' and 'testbed' but does not provide specific dataset split information (exact percentages, sample counts, or detailed splitting methodology) for training, validation, and testing. |
| Hardware Specification | No | The paper does not provide specific hardware details (exact GPU/CPU models, processor types, or memory amounts) used for running its experiments. It only mentions 'modest hardware' in a general sense. |
| Software Dependencies | No | The paper mentions using 'Sym Py' as a symbolic solver and that Python code is generated, but it does not provide specific version numbers for Sym Py, Python, or any other relevant software libraries. |
| Experiment Setup | Yes | Details of the hyperparameters used in SYRELM are described in Appendix B. |