Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in Coakley et alK. L. Coakley, T. Snelleman, H. Hoos, and O. E. Gundersen, "The embrace of open science: An analysis of a decade of AI research and 56 800 conference papers," Under Review, 2026..
SUMO: Subspace-Aware Moment-Orthogonalization for Accelerating Memory-Efficient LLM Training
Authors: Yehonathan Refael, Guy Smorodinsky, Tom Tirer, Ofir Lindenbaum
NeurIPS 2025 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Empirical evaluations confirm that SUMO accelerates convergence, enhances stability, improves performance, and reduces memory requirements by up to 20% compared to state-of-the-art methods. Empirical evaluations confirm that SUMO accelerates convergence, enhances stability, improves performance, and reduces memory requirements by up to 20% compared to state-of-the-art methods. Empirical evaluations confirm that SUMO accelerates convergence, enhances stability, improves performance, and reduces memory requirements by up to 20% compared to state-of-the-art methods. |
| Researcher Affiliation | Academia | Yehonathan Refael Faculty of Engineering Tel Aviv University EMAIL, Guy Smorodinsky Department of Computer science Ben Gurion University EMAIL, Tom Tirer Faculty of Engineering Bar-Ilan University EMAIL, Ofir Lindenbaum Faculty of Engineering Bar-Ilan University EMAIL |
| Pseudocode | Yes | Algorithm 1 SUMO: Subspace-Aware Moment-Orthogonalization Optimization |
| Open Source Code | Yes | Code: https://github.com/guy120494/SUMO. |
| Open Datasets | Yes | Our model was evaluated using the GLUE benchmark [43] through the fine-tuning of the pre-trained Roberta-base model [42] across eight tasks. To highlight the effectiveness of our method in pre-training, we pre-train Llama models following the evaluation protocol of [1] and compare performance with the state-of-the-art method, in terms of perplexity and memory usage. For this evaluation, we trained large Llama-based models on the C4 dataset, a curated and extensive version of the Common Crawl web corpus [46]. To evaluate the performance of our method on a complex reasoning task, we utilize the GSM8K dataset [47] to test systematic generalization. |
| Dataset Splits | Yes | Fine-tuning on GLUE benchmark. Pre-training Llama on C4 Dataset. To evaluate the performance of our method on a complex reasoning task, we utilize the GSM8K dataset [47] to test systematic generalization. For these experiments, we used a batch size of 32 and 10 epochs for fine-tuning. Tables 10 and 11 also specify hyperparameters for fine-tuning RoBERTa-base, including Batch Size, # Epochs, Max Seq. Len. |
| Hardware Specification | Yes | The experiments were carried out using the NVIDIA A100 GPU. Experiments were conducted using an NVIDIA H200 GPU. |
| Software Dependencies | No | The paper does not provide specific version numbers for software dependencies such as Python, PyTorch, or other libraries used in the experiments. |
| Experiment Setup | Yes | Table 10: Hyperparameters of fine-tuning Ro BERTa base for the comparison in Table 2 with respect only to rank=4. Table 11: Hyperparameters of fine-tuning Ro BERTa base for the comparison in Table 2 with respect only to rank=8. Table 9: Perplexity from a grid-search over Subspace Update Frequency (K) and Rank (r) for LLa MA-130M on C4. |