Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in Coakley et alK. L. Coakley, T. Snelleman, H. Hoos, and O. E. Gundersen, "The embrace of open science: An analysis of a decade of AI research and 56 800 conference papers," Under Review, 2026..
BOBCAT: Bilevel Optimization-Based Computerized Adaptive Testing
Authors: Aritra Ghosh, Andrew Lan
IJCAI 2021 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Through extensive experiments on five real-world student response datasets, we show that BOBCAT outperforms existing CAT methods (sometimes significantly) at reducing test length. |
| Researcher Affiliation | Academia | Aritra Ghosh and Andrew Lan University of Massachusetts Amherst EMAIL |
| Pseudocode | Yes | Algorithm 1 BOBCAT training process |
| Open Source Code | Yes | Our implementation will be publicly available at https://github.com/arghosh/BOBCAT. |
| Open Datasets | Yes | We use five publicly available benchmark datasets: Ed Net2, Junyi3, Eedi-1, Eedi-24, and ASSISTments5. ... 2https://github.com/riiid/ednet 3https://www.kaggle.com/junyiacademy/learning-activitypublic-dataset-by-junyi-academy 4https://eedi.com/projects/neurips-education-challenge 5https://sites.google.com/site/assistmentsdata/home/assistment2009-2010-data |
| Dataset Splits | Yes | We perform 5-fold cross validation for all datasets; for each fold, we use 60%-20%-20% students for training, validation, and testing, respectively. |
| Hardware Specification | Yes | We implement all methods in Py Torch and run our experiments in a NVIDIA Titan X/1080Ti GPU. |
| Software Dependencies | No | The paper mentions 'Py Torch' but does not specify a version number for it or other software dependencies. |
| Experiment Setup | Yes | For Bi NN, we use a two-layer, fully-connected network (with 256 hidden nodes, Re LU nonlinearity, 20% dropout rate, and a final sigmoid output layer)... We use another fully-connected network (with two hidden layers, 256 hidden nodes, Tanh nonlinearity, and a final softmax output layer) as the question selection algorithm. |