Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in Coakley et alK. L. Coakley, T. Snelleman, H. Hoos, and O. E. Gundersen, "The embrace of open science: An analysis of a decade of AI research and 56 800 conference papers," Under Review, 2026..
Metritocracy: Representative Metrics for Lite Benchmarks
Authors: Ariel D Procaccia, Ben Schiffer, Serena Wang, Shirley Zhang
NeurIPS 2025 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Finally, we tie theory to practice through real-world case studies on both LLM evaluation and hospital quality evaluation. |
| Researcher Affiliation | Academia | Ariel D. Procaccia Harvard University Benjamin Schiffer Harvard University Serena Wang Harvard University Shirley Zhang Harvard University |
| Pseudocode | Yes | Algorithm 1 Greedy (pseudo-code) |
| Open Source Code | Yes | Code is provided in the Supplemental Materials to reproduce all experiments. |
| Open Datasets | Yes | Data was accessed using the Big-bench API available at https://github.com/google/BIG-bench. |
| Dataset Splits | No | The paper describes using existing datasets like BIG-bench, HELM, and CMS Hospital Compare, defining 'alternatives' (LLMs, hospitals) and 'metrics' (evaluation measures). It does not, however, detail specific train/test/validation splits for these datasets as part of its own experimental methodology, as its focus is on selecting representative subsets of metrics rather than training predictive models. |
| Hardware Specification | Yes | All experiments were run on a Mac Book Pro with an Intel Core i7. |
| Software Dependencies | No | Integer programs were solved using CPLEX. |
| Experiment Setup | Yes | We evaluate each method by comparing the subset sizes |K| achieved for each tolerance parameter (group size g for PR, tolerance ϵ for PP)." and "A maximum solve time of 10 minutes was set for each integer program. |