Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in [1].
Transcendence: Generative Models Can Outperform The Experts That Train Them
Authors: Edwin Zhang, Vincent Zhu, Naomi Saphra, Anat Kleiman, Benjamin Edelman, Milind Tambe, Sham Kakade, Eran Malach
NeurIPS 2024 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We demonstrate transcendence by training an autoregressive transformer to play chess from game transcripts, and show that the trained model can sometimes achieve better performance than all players in the dataset.1 We theoretically prove that transcendence can be enabled by low-temperature sampling, and rigorously assess this claim experimentally. |
| Researcher Affiliation | Collaboration | Edwin Zhang Open AI Harvard University Humanity Unleashed EMAIL Vincent Zhu UC Santa Barbara Humanity Unleashed EMAIL Naomi Saphra Harvard University Kempner Institute EMAIL Anat Kleiman Harvard University Apple EMAIL Benjamin L. Edelman Princeton University Harvard University EMAIL Milind Tambe Harvard University EMAIL Sham Kakade Harvard University Kempner Institute EMAIL Eran Malach Harvard University Kempner Institute EMAIL |
| Pseudocode | No | The paper includes theoretical proofs in the appendix but does not contain any structured pseudocode or algorithm blocks. |
| Open Source Code | Yes | To play with our models, code, and data, please see our website at https://transcendence.eddie.win. We also release our code openly to support further research into transcendence... |
| Open Datasets | Yes | Our dataset consists of human chess games from the lichess.org open source database from January 2023 to October 2023. We use the Glicko-2 rating system [7], which is also adopted by https://lichess.org, the free and open-source online chess server from which we source our dataset. |
| Dataset Splits | No | The paper discusses training on datasets and testing against Stockfish, but does not explicitly state the training, validation, and test dataset splits with percentages or counts. |
| Hardware Specification | Yes | We train all of our models on the Nvidia H100 80GB GPU. |
| Software Dependencies | No | The paper mentions 'Adam W [13]' for the optimizer and refers to 'modern large model training' and the OPT-175B team's practices, but does not provide specific version numbers for software dependencies like Python, PyTorch, or other libraries. |
| Experiment Setup | Yes | We give a full list of the hyperparameters we used for training here. |