Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in Coakley et alK. L. Coakley, T. Snelleman, H. Hoos, and O. E. Gundersen, "The embrace of open science: An analysis of a decade of AI research and 56 800 conference papers," Under Review, 2026..
Distilling LLM Agent into Small Models with Retrieval and Code Tools
Authors: Minki Kang, Jongwon Jeong, Seanie Lee, Jaewoong Cho, Sung Ju Hwang
NeurIPS 2025 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We evaluate our method on eight reasoning tasks across factual and mathematical domains, covering both in-domain and out-of-domain generalization. Our results show that s LMs as small as 0.5B, 1.5B, 3B parameters can achieve performance competitive with nexttier larger 1.5B, 3B, 7B models fine-tuned using Co T distillation, demonstrating the potential of agent distillation for building practical, tool-using small agents. |
| Researcher Affiliation | Collaboration | Minki Kang1 Jongwon Jeong2 Seanie Lee1 Jaewoong Cho3 Sung Ju Hwang1,4 1KAIST, 2University of Wisconsin-Madison, 3KRAFTON, 4Deep Auto.ai EMAIL |
| Pseudocode | No | The paper describes methods like 'first-thought prefix' and 'self-consistent action generation' and provides conceptual diagrams (Figure 3), but it does not contain explicit pseudocode or algorithm blocks with structured steps. |
| Open Source Code | Yes | Our code is available at https://github.com/Nardien/agent-distillation. |
| Open Datasets | Yes | We evaluate our proposed Agent Distillation across benchmarks to test whether small language models (s LMs) can acquire agentic abilities from a large language model (LLM) agent teacher. ... We use 1,000 Hot Pot QA [2] and 2,000 MATH [6] examples for training. For test benchmarks, we summarize them in Table 1. |
| Dataset Splits | Yes | We use 1,000 Hot Pot QA [2] and 2,000 MATH [6] examples for training. For test benchmarks, we summarize them in Table 1. To reduce evaluation cost, we limit each test set to 500 examples, following Wang et al. [71]. ... Specifically, we only use 1000 hard examples from Hot Pot QA and 1000 level 2-3 examples, 1000 level 4-5 examples from MATH dataset. |
| Hardware Specification | Yes | All experiments are conducted using four NVIDIA A100 80GB GPUs. |
| Software Dependencies | No | The paper mentions 'Lo RA (rank 64)' as a tuning technique and 'e5-base-v2' as embeddings, and 'smolagents' library as an agent framework. However, it does not provide specific version numbers for these or any other software components. |
| Experiment Setup | Yes | All models are fine-tuned for 2 epochs using a batch size of 8 and a learning rate of 2 10 4. ... For inference, we use a greedy decoding. For all agents, we set max steps to 5. For sag in main experiments, we set N = 8 with temperature to 0.4. More details are in Appendix C. |