Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in Coakley et alK. L. Coakley, T. Snelleman, H. Hoos, and O. E. Gundersen, "The embrace of open science: An analysis of a decade of AI research and 56 800 conference papers," Under Review, 2026..
Music Recommenders: User Evaluation Without Real Users?
Authors: Susan Craw, Ben Horsburgh, Stewart Massie
IJCAI 2015 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Experiments show that its findings are consistent with those from an online study with real users. This paper explores system-centric evaluation based on user data from social media sites relevant to music. |
| Researcher Affiliation | Academia | Susan Craw and Ben Horsburgh and Stewart Massie IDEAS Research Institute and School of Computing Science & Digital Media Robert Gordon University, Aberdeen, UK EMAIL |
| Pseudocode | No | The paper does not contain any structured pseudocode or algorithm blocks. |
| Open Source Code | No | The paper does not provide any statement or link indicating that the source code for the described methodology is publicly available. |
| Open Datasets | Yes | The Million Song Dataset [MSD, nd; Bertin-Mahieux et al., 2011] is a step in the right direction, but provides audio data as pre-extracted features, and so cannot be used to evaluate recommenders based on other audio features. Our music collection is Horsburgh s [2013] dataset containing 3174 audio tracks by 764 separate artists. |
| Dataset Splits | No | The paper does not specify explicit training, validation, and test dataset splits with percentages or sample counts for model training. It describes a 'user study' as a 'benchmark to validate' the social Sim evaluation, but this is a conceptual validation, not a data split for model training. |
| Hardware Specification | No | The paper does not provide any specific details about the hardware (e.g., CPU, GPU models, memory, or cloud instances) used for running the experiments. |
| Software Dependencies | No | The paper mentions software like 'Last.fm s API' and 'Vamp plugin' but does not provide specific version numbers for these or any other software dependencies, which would be necessary for reproducible setup. |
| Experiment Setup | Yes | A k-NN nearest-neighbour retrieval using cosine similarity in the MFS-LSI space identifies the K = 40 most similar audio tracks. The number of pseudo-tags P from p to be included in the hybrid vector h, to balance the number of existing tags T in t. A weighting α determines the influence of selected pseudo-tags p on h1. h = α p + (1 α)t where P = 100 T α = 0.5 P/100 |