A Unified Adaptive Testing System Enabled by Hierarchical Structure Search
Authors: Junhao Yu, Yan Zhuang, Zhenya Huang, Qi Liu, Xin Li, Rui Li, Enhong Chen
ICML 2024 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | In this section, we conduct qualitative and quantitative analyses on our UATS framework . This evaluation was carried out on three real-world datasets to assess the effectiveness of our approach. |
| Researcher Affiliation | Academia | 1State Key Laboratory of Cognitive Intelligence, University of Science and Technology of China 2Institute of Artificial Intelligence, Hefei Comprehensive National Science Center 3School of Computer Science and Technology, Xidian University. |
| Pseudocode | Yes | Algorithm 1 UATS Framework Training Process |
| Open Source Code | Yes | The specific code can be found at: https://github.com/bigdata-ustc/UATS. We will provide continuous maintenance. |
| Open Datasets | Yes | Datasets. We worked with three distinct sets of educational data: the ASSIST dataset (Pardos et al., 2013), which comprises student practice logs related to mathematics and knowledge concepts problems from the ASSISTments online tutoring system. The NIPS-EDU dataset (Wang et al., 2021) originated from the Neur IPS 2020 Education Challenge and consists of student responses to questions on the NIPS-EDU educational platform. |
| Dataset Splits | Yes | We conducted 5-fold cross-validation on all datasets. For each fold, we allocated 60% of the students for training, 20% for validation, and 20% for testing. |
| Hardware Specification | Yes | All experiments were run on an NVIDIA V100 GPU. |
| Software Dependencies | No | The paper mentions cognitive diagnosis models like 'Item Response Theory (IRT)' and 'Neural Cognitive Diagnostic Model (Neural CDM)', and implicitly uses frameworks given the GitHub link, but it does not provide specific version numbers for any software dependencies or libraries required for reproduction. |
| Experiment Setup | No | The paper refers to learning rates γ and β in its theoretical sections (Algorithm 1, Theorem 4.4) but does not provide specific numerical values for these or any other hyperparameters (like batch size, epochs, or optimizer configuration) in the experimental settings section. |