Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in Coakley et alK. L. Coakley, T. Snelleman, H. Hoos, and O. E. Gundersen, "The embrace of open science: An analysis of a decade of AI research and 56 800 conference papers," Under Review, 2026..
OSTAR: Optimized Statistical Text-classifier with Adversarial Resistance
Authors: Yuhan Yao, Feifei Kou, Lei Shi, Xiao yang, Zhongbao Zhang, Suguo Zhu, Jiwei Zhang, Lirong Qiu, LI Haisheng
NeurIPS 2025 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Experimental results on three public datasets under various adversarial scenarios demonstrate that our framework outperforms existing MGT detection methods, achieving state-of-the-art performance and robust against attacks. |
| Researcher Affiliation | Academia | Yuhan Yao1,2, Feifei Kou1,2 , Lei Shi3, Xiao Yang1, Zhongbao Zhang1, Suguo Zhu4 Jiwei Zhang1, Lirong Qiu1,2, Haisheng Li5 1 School of Computer Science (National Pilot School of Software Engineering), BUPT 2 Key Laboratory of Trustworthy Distributed Computing and Service, BUPT, Ministry of Education 3State Key Laboratory of Media Convergence and Communication, CUC 4College of Computer Science and Technology, HDU 5School of Computer and Artificial Intelligence, BTBU *Correspondence: EMAIL |
| Pseudocode | Yes | Algorithm 1 OSTAR train process |
| Open Source Code | Yes | The code is available at https://github.com/BUPT-SN/OSTAR. |
| Open Datasets | Yes | In this study, we employed three widely-used and moderately challenging datasets: Check GPT[41], HC3[29], and a cross-domain dataset generated by GLM-130B from the Deep Fake[33]. |
| Dataset Splits | Yes | The detailed composition of our original datasets is presented in Table 4, while the configuration of the attacked datasets (generated via perturbation/paraphrases attacks) is summarized in Table 5. The two-tuple (human, machine) in the table represents the number of human texts and the number of AI texts. The symbol "*" in Table 5 indicates that nine distinct adversarial perturbation methods (shown in appendix E) were applied to each original sample, resulting in a tenfold expansion of the dataset size. To mitigate computational overhead caused by this exponential growth, we selected part of each dataset to include 500 human-authored and 500 machine-generated samples for balanced training and testing. Table 6 shows the detailed composition of the attacked datasets used in the Evaluation on Attacked Datasets experiment, where the " * " symbol indicates the multiplication factor applied to the original sample counts due to adversarial attacks. |
| Hardware Specification | Yes | Our model was trained on an NVIDIA RTX 3090 GPU, requiring approximately 17GB of VRAM with a batch size of 4, making it feasible to implement under most laboratory conditions. |
| Software Dependencies | No | Our implementation uses RoBERTa as the base pretrained model. Identical attack procedures were applied to both training and test sets. For the MDSP, we set length l = 3. The contrastive learning component employs a weight of 0.02 and temperature coefficient of 0.05. Training utilizes the Adam optimizer with learning rate 1 * 10^-5 and adam epsilon of 1 * 10^-8. Our model was trained on an NVIDIA RTX 3090 GPU, requiring approximately 17GB of VRAM with a batch size of 4, making it feasible to implement under most laboratory conditions. |
| Experiment Setup | Yes | For the MDSP, we set length l = 3. The contrastive learning component employs a weight of 0.02 and temperature coefficient of 0.05. Training utilizes the Adam optimizer with learning rate 1 * 10^-5 and adam epsilon of 1 * 10^-8. Our model was trained on an NVIDIA RTX 3090 GPU, requiring approximately 17GB of VRAM with a batch size of 4, making it feasible to implement under most laboratory conditions. |