Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in Coakley et alK. L. Coakley, T. Snelleman, H. Hoos, and O. E. Gundersen, "The embrace of open science: An analysis of a decade of AI research and 56 800 conference papers," Under Review, 2026..
WEDGE: Synthesizing Performance Constraints for Evaluating and Improving Code Efficiency
Authors: Jun Yang, Cheng-Chi Wang, Bogdan "Bo" Stoica, Kexin Pei
NeurIPS 2025 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Our extensive evaluation shows that WEDGE introduces a significant slowdown compared to the tests in Code Contests and those claimed to be optimized by existing approaches. From the utility perspective, integrating our tests substantially improves the existing code optimization approaches that rely on test-driven execution feedback. |
| Researcher Affiliation | Academia | Jun Yang, Cheng-Chi Wang, Bogdan Alexandru Stoica, Kexin Pei EMAIL Department of Computer Science, The University of Chicago |
| Pseudocode | No | The paper describes the workflow of WEDGE in Figure 1 and its components in text, and provides code listings (e.g., Listing 1-13 for examples and constraints) but does not include any explicitly labeled pseudocode or algorithm blocks for the methodology. |
| Open Source Code | Yes | We release PERFFORGE, the performance tests generated by WEDGE, to benchmark future approaches for efficient code generation at https://github.com/UChi Seclab/perfforge. |
| Open Datasets | Yes | We release PERFFORGE, the performance tests generated by WEDGE, to benchmark future approaches for efficient code generation at https://github.com/UChi Seclab/perfforge. We evaluate WEDGE on Code Contests [32] with a wide range of competitive programming problems and human-written solutions. |
| Dataset Splits | Yes | We rank the problems based on the coefficient of variation [12] of the CPU instruction counts and select the top 300 problems. For each solution, we run EFFI-LEARNER with three different prompts to let EFFI-LEARNER optimize the code. We then select the five slowest ones per problem to form our evaluation dataset. Ultimately, we end up with 56 problems and 280 Python solutions. |
| Hardware Specification | Yes | We run experiments on six x86-64 machines equipped with a 24-core Intel Xeon Gold 6126 CPU with 192GB of RAM. |
| Software Dependencies | Yes | Each machine runs Ubuntu 20.04 LTS (kernel version 5.4.0). We use the Open AI GPT-4o (gpt-4o-2024-08-06) and Deep Seek V3 (deepseek-v3-2024-12-26) with a temperature of 0.8 and max_length of 4,096. We use AFL++ as its fuzzing engine. We use Python as our main development language and rely on the perf and gcov Linux utilities to collect instruction count, physical, and code coverage metrics. |
| Experiment Setup | Yes | We use the Open AI GPT-4o (gpt-4o-2024-08-06) and Deep Seek V3 (deepseek-v3-2024-12-26) with a temperature of 0.8 and max_length of 4,096. To further reduce the noise, we average the CPU instructions over five runs for each program throughout all experiments. To collect inputs, We run WEDGE s fuzzing (based on our modified AFL++) for one hour for each solution in parallel. |