JAG: A Crowdsourcing Framework for Joint Assessment and Peer Grading
Authors: Igor Labutov, Christoph Studer
AAAI 2017 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We demonstrate the effectiveness and limits of our framework via simulations and a real-world user study. |
| Researcher Affiliation | Academia | Igor Labutov Carnegie Mellon University Pittsburgh, PA 15213 Christoph Studer Cornell University Ithaca, NY 14853 |
| Pseudocode | No | The paper describes the EM algorithm in text and mathematical equations but does not include a formal pseudocode or algorithm block. |
| Open Source Code | No | The paper does not contain any explicit statement about releasing source code or a link to a code repository. |
| Open Datasets | No | The paper describes data collected via Amazon Mechanical Turk, but it does not provide concrete access information (e.g., a specific link, DOI, repository name, or formal citation with authors/year) for this dataset or any other publicly available dataset used. |
| Dataset Splits | No | The paper does not specify exact training, validation, and test split percentages or sample counts, nor does it reference predefined splits with citations for reproducibility. |
| Hardware Specification | No | The paper mentions 'Computational resources were sponsored in part by grants from Amazon and Microsoft' in the acknowledgements, suggesting cloud usage. However, it does not specify any exact GPU/CPU models, processor types, or detailed computer specifications used for running its experiments. |
| Software Dependencies | No | The paper mentions using 'the L-BFGS algorithm (Zhu et al. 1997)' but does not list any specific software components with their version numbers (e.g., Python, PyTorch, or specific library versions) that are critical for replicating the experiment. |
| Experiment Setup | No | The paper provides details on the synthetic data generation parameters (e.g., |Sopen|, |Smcq|, |Q|) but does not explicitly state specific hyperparameters (e.g., learning rate, batch size, number of epochs, optimizer settings) or other detailed system-level training configurations for either the synthetic or real-world experiments. |