Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in Coakley et alK. L. Coakley, T. Snelleman, H. Hoos, and O. E. Gundersen, "The embrace of open science: An analysis of a decade of AI research and 56 800 conference papers," Under Review, 2026..
SuffixDecoding: Extreme Speculative Decoding for Emerging AI Applications
Authors: Gabriele Oliaro, Zhihao Jia, Daniel Campos, Aurick Qiao
NeurIPS 2025 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Evaluations on agentic benchmarks, including SWE-Bench and Text-to-SQL, demonstrate that Suffix Decoding achieves speedups of up to 5.3 , outperforming state-of-the-art methods 2.8 faster than model-based approaches like EAGLE-2/3 and 1.9 faster than model-free approaches such as Token Recycling. Suffix Decoding is open-sourced at https://github.com/snowflakedb/Arctic Inference. |
| Researcher Affiliation | Collaboration | Gabriele Oliaro , Zhihao Jia Daniel Campos Aurick Qiao Snowflake AI Research Carnegie Mellon University EMAIL, EMAIL |
| Pseudocode | Yes | Algorithm 1 Speculation Tree Generation function EXPANDSPECULATIONTREE(N_p, MAX_SPEC) |
| Open Source Code | Yes | Suffix Decoding is open-sourced at https://github.com/snowflakedb/Arctic Inference. |
| Open Datasets | Yes | We evaluate on both agentic and non-agentic workloads. For agentic applications, we trace requests from two real applications: Open Hands [Wang et al., 2024c] on SWE-Bench [Jimenez et al., 2024] (a Git Hub issue resolution benchmark), and Agentic SQL, a proprietary multi-agent pipeline application for SQL generation. For non-agentic workloads, we use Spec-Bench [Xia et al., 2024], consisting of openended, single-turn tasks across 13 categories, in-cluding 8 MT-Bench categories (Writing, Roleplay, Reasoning, Math, Coding, Extraction, STEM, Humanities). We also used Wild Chat [Zhao et al., 2024], Magicoder [Wei et al., 2023], and Spider [Yu et al., 2018]. |
| Dataset Splits | No | The paper uses several benchmarks like SWE-Bench and Spec-Bench, which usually come with predefined splits. However, the paper itself does not explicitly state the training/test/validation splits used for these datasets, nor does it describe how any custom datasets were split. |
| Hardware Specification | Yes | We conducted our experiments on a single p5.48xlarge AWS instance equipped with 8 NVIDIA H100 80G GPUs and 2TB of main memory. |
| Software Dependencies | No | First, we updated the code to work with the latest version of the transformers library, which is required to run recent open-source LLMs such as meta-llama/Llama-3.1. We also added support for arbitrary datasets (such as SWE-Bench and Agentic SQL) and implemented Suffix Decoding within the framework. We implemented Suffix Decoding in v LLM [Kwon et al., 2023]. We used the flashinfer kernels for sampling. |
| Experiment Setup | Yes | In practice, we found that α [1, 4] works well for agentic applications. Table 3 shows the wall-clock speedup results on Spec-Bench for various threshold values. For open-ended generation tasks, we find that setting τ close to or slightly exceeding the mean accepted tokens (MAT) of the model-based speculator yields the best performance. For instance, EAGLE-3 achieves a MAT of approximately 4.65 tokens/step on Spec-Bench. The results show that τ [5, 7] produces the highest overall speedups (2.5 ), with performance being relatively stable across this range. We served the all-hands/openhands-lm-32b-v0.1-ep3 model locally using v LLM, with a tensor parallelism degree of 4 and with prefix caching enabled. We ran Open Hands with the Code Act Agent [Wang et al., 2024b] with ITERATIVE_EVAL_MODE=true, and a maximum of 100 iterations, as recommended by the Open Hands authors. We used a maximum of 16 concurrent workers to run the SWE-Bench tasks. |