Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in Coakley et alK. L. Coakley, T. Snelleman, H. Hoos, and O. E. Gundersen, "The embrace of open science: An analysis of a decade of AI research and 56 800 conference papers," Under Review, 2026..
Q3R: Quadratic Reweighted Rank Regularizer for Effective Low-Rank Training
Authors: Ipsita Ghosh, Ethan Nguyen, Christian Kümmerle
NeurIPS 2025 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Numerical experiments show that Q3R is able to reduce the number of parameters in Vi T models by 60% during pre-training on CIFAR-10, with only around 1.3% accuracy drop. We validate the performance of Q3R for low-rank fine-tuning with experiments fine-tuning Ro BERTa and Llama3 on GLUE tasks, for which Q3R achieves comparable performance compared to dense fine-tuning and state-of-the art low-rank PEFT methods. |
| Researcher Affiliation | Collaboration | Ipsita Ghosh Department of Computer Science University of Central Florida EMAIL Ethan Nguyen Department of Computer Science University of North Carolina at Charlotte EMAIL Christian Kümmerle School of Data, Mathematical and Statistical Sciences Department of Computer Science University of Central Florida EMAIL |
| Pseudocode | Yes | Algorithm 1 Update Reweighting Operator RW1,ϵoldp q ÞÑ RW,ϵnewp q Algorithm 2 Low-Rank Training via Adam Q3R Algorithm 3 Computation of the Q3R function value Q3RW1,ϵp Wq Algorithm 4 COMPUTATION GRADIENT OF Q3R : Compute RW1,ϵp Wq |
| Open Source Code | Yes | The code is available at https://github.com/That E10/q3r.git. |
| Open Datasets | Yes | We pre-train Vi T-Tiny (DBKea21; SKZ 22) on CIFAR-10, Vi T-Base (DBKea21) on CIFAR-100, followed by post-training low-rank truncation (AZW); further, we fine-tune BERT-Large (DCLT19) on GLUE benchmark tasks (without truncation). |
| Dataset Splits | Yes | We fine-tuned pre-trained Ro BERTa models on the GLUE benchmark using Adam Q3R... MRPC: 3.7 k train / 1.7 k test; paraphrase detection; accuracy / F1 SST-2: 67 k train / 1.8 k test; sentiment classification; accuracy STS-B: 7 k train / 1.4 k test; sentence similarity; Pearson / Spearman correlation QQP: 364 k train / 391 k test; question paraphrase detection; accuracy / F1 MNLI: 393 k train / 20 k matched + 20 k mismatched test; entailment classification; accuracy QNLI: 105 k train / 5.4 k test; question answer entailment; accuracy RTE: 2.5 k train / 3 k test; textual entailment; accuracy WNLI: 634 train / 146 test; coreference-based inference; accuracy |
| Hardware Specification | Yes | Training is conducted with a learning rate of α 4 ˆ 10 5, a batch size of 384, and gradient clipping (ZHSJ20) across 4 L40S GPUs. For few experiments like Q3R, we used NVIDIA A5000 to train the Vi T models. The rest of the experiments were performed on NVIDIA V100 with 32GB memory. The fine-tuning experiemtns were all performed in NVIDIA A5000 GPUs. |
| Software Dependencies | No | The paper mentions deep learning frameworks like Transformers and optimizers like Adam, but does not specify exact version numbers for any software dependencies like Python, PyTorch, CUDA, or Hugging Face libraries. It describes its own Adam Q3R optimizer, but that's part of the methodology, not an ancillary software dependency with a version. |
| Experiment Setup | Yes | We train Vi T-Tiny on CIFAR-10 for 100 epochs using a learning rate of α 0.00004. Input: Minibatch size B, reweighting period T, Q3R parameter λ, learning rate α 0.001, β1 0.9, β2 0.999, δ 10 8, η 3, target rank rtarget. |