simpleKT: A Simple But Tough-to-Beat Baseline for Knowledge Tracing
Authors: Zitao Liu, Qiongqiong Liu, Jiahao Chen, Shuyan Huang, Weiqi Luo
ICLR 2023 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Extensive experiments show that such a simple baseline is able to always rank top 3 in terms of AUC scores and achieve 57 wins, 3 ties and 16 loss against 12 DLKT baseline methods on 7 public datasets of different domains. |
| Researcher Affiliation | Collaboration | Zitao Liu Guangdong Institute of Smart Education, Jinan University, Guangzhou, China liuzitao@jnu.edu.cn Qiongqiong Liu, Jiahao Chen, Shuyan Huang TAL Education Group, Beijing, China {liuqiongqiong1, chenjiahao, huangshuyan}@tal.com Weiqi Luo Guangdong Institute of Smart Education, Jinan University, Guangzhou, China lwq@jnu.edu.cn |
| Pseudocode | No | The paper describes the proposed method using mathematical formulations and descriptive text, but does not include a dedicated pseudocode or algorithm block. |
| Open Source Code | Yes | Code is available at https://github.com/pykt-team/pykt-toolkit1. To encourage reproducible research, all the related codes, data and the learned SIMPLEKT models are publicly available at https://github.com/pykt-team/pykt-toolkit. The code of SIMPLEKT and its variants, i.e., SIMPLEKT-Scalar Diff and SIMPLEKT-No Diff, to reproduce the experimental results can be found at https://github.com/pykt-team/ pykt-toolkit. |
| Open Datasets | Yes | In this paper, we experiment with 7 widely used datasets to comprehensively evaluate the performance of our models. ... ASSISTments2009 (AS2009)6: ... https://sites.google.com/site/assistmentsdata/home/2009-2010-assistment-data/ skill-builder-data-2009-2010. Algebra2005 (AL2005)7: ... https://pslcdatashop.web.cmu.edu/KDDCup/. Bridge2006 (BD2006)7: ... https://pslcdatashop.web.cmu.edu/KDDCup/. NIPS348: ... https://eedi.com/projects/neurips-education-challenge. Statics20119: ... https://pslcdatashop.web.cmu.edu/Dataset Info?dataset Id=507. ASSISTments2015 (AS2015)10: ... https://sites.google.com/site/assistmentsdata/datasets/2015-assistments-skill-builder-data. POJ11: ... https://drive.google.com/drive/folders/1LRljq Wf ODw TYRMPw6w EJ_m Mt1KZ4x BDk. |
| Dataset Splits | Yes | Similar to (Liu et al., 2022), we randomly withhold 20% of the students sequences for model evaluation and we perform standard 5-fold cross validation on the rest 80% of each dataset. |
| Hardware Specification | Yes | Our model is implemented in Py Torch and trained on NVIDIA RTX 3090 GPU device. |
| Software Dependencies | No | The paper states 'Our model is implemented in Py Torch' but does not specify a version number for PyTorch or any other software dependencies. |
| Experiment Setup | Yes | The embedding dimension, the hidden state dimension, the two dimension of the prediction layers are set to [64, 128], the learning rate and dropout rate are set to [1e-3, 1e-4, 1e-5] and [0.05, 0.1, 0.3, 0.5] respectively, the number of blocks and attention heads are set to [1, 2, 4] and [4, 8], the seed is set to [42, 3407] for reproducing the experimental results. |