Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in [1].
Novel Spectral Algorithms for the Partial Credit Model
Authors: Duc Nguyen, Anderson Ye Zhang
ICML 2024 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | In this section, we present the empirical findings to complement our theoretical contribution and showcase the usefulness of the spectral algorithm. The detailed description of the experiment is deferred to the supplementary materials. We compare the spectral algorithm against two well known and popular PCM inference algorithms the marginal likelihood estimate (MMLE) (Basu, 2011) and the joint maximum likelihood estimate (JMLE) (Andersen, 1973; Fischer, 1981; Haberman, 1977). |
| Researcher Affiliation | Academia | 1Department of Computer & Information Science, University of Pennsylvania 2Department of Statistics & Data Science, Wharton School, University of Pennsylvania. |
| Pseudocode | Yes | Algorithm 1 Spectral Algorithm, Algorithm 2 Parameter Shift Estimation Algorithm, Algorithm 3 The Spectral-EM Algorithm |
| Open Source Code | Yes | The implementation of our algorithms can be found at https://github.com/dnguyen1196/spectral-algos-discrete-data. |
| Open Datasets | Yes | Table 3: Datasets metadata and references. LSAT (Mc Donald, 2013), UCI (Hussain et al., 2018), GRADES (Cortez & Silva, 2008), HETREC (Cantador et al., 2011), EACH MOVIE (Harper & Konstan, 2015), ML-1M (Harper & Konstan, 2015), ML-10M (Harper & Konstan, 2015), ML-20M (Harper & Konstan, 2015), BOOK-GENOME (Kotkov et al., 2022). |
| Dataset Splits | Yes | For each of the remaining users, we leave out one rating from each user as part of the heldout test dataset and one rating as part of a validation dataset (for MMLE and Spectral-EM). |
| Hardware Specification | No | The paper does not explicitly describe the specific hardware used for running its experiments. |
| Software Dependencies | No | The paper mentions using specific implementations like 'MMLE and JMLE can be found in Sanchez (2021)' and provides a link to their own implementation, but it does not specify versions for general software dependencies (e.g., programming languages, libraries, frameworks) required for reproducibility. |
| Experiment Setup | Yes | In each trial, we generate Īøl N(0, Ļ0) for either Ļ0 = 1 (Figure 1) or Ļ0 = 2 (Figure 3). In all experiments, the prior distribution used in MMLE is set to be the standard normal distribution. The validation dataset is used by MMLE to select the best user parameter distribution DĪø and by Spectral-EM to select the number of mixture components. We set 2011-01 to 2018-01 as the training period where we find the model with the appropriate hyper-parameters that maximizes the PNL within the training period. |