Evaluating Recommender System Stability with Influence-Guided Fuzzing
Authors: David Shriver, Sebastian Elbaum, Matthew B. Dwyer, David S. Rosenblum4934-4942
AAAI 2019 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We implement our approach and evaluate it on several recommender algorithms using the Movie Lens dataset. We find that influence-guided fuzzing can effectively find small sets of modifications that cause significantly more instability than random approaches. |
| Researcher Affiliation | Academia | 1Department of Computer Science, University of Virginia, Charlottesville, VA, USA 2Department of Computer Science, National University of Singapore, Singapore |
| Pseudocode | No | The paper defines functions and heuristics mathematically but does not include structured pseudocode or algorithm blocks. |
| Open Source Code | No | The paper provides a link (https://github.com/lyst/lightfm) for the Light FM algorithm, which is a third-party tool used in their study, not their own source code for the methodology described in the paper. |
| Open Datasets | Yes | Movie Lens 100k Dataset We use the dataset released in 1998 from the Movie Lens recommendation system. The dataset is available as a group of tab separated files, containing 100 thousand integer ratings (from 1 to 5) collected from 943 users over a period of 8 months on 1682 movies. Each user has rated at least 20 items. More details about the data collection process and the dataset itself are available at https://grouplens.org/datasets/movielens/100k/ (Harper and Konstan 2015). |
| Dataset Splits | No | The paper mentions modifying the Movie Lens 100k dataset and using it for training, but it does not specify any explicit train/validation/test splits of the original dataset for model training or evaluation in the standard sense (e.g., 80/10/10 split). |
| Hardware Specification | No | The paper does not provide any specific details about the hardware used to run the experiments, such as GPU models, CPU types, or memory specifications. |
| Software Dependencies | No | The paper mentions using the 'Lenskit framework' and a 'Python implementation' for Light FM, but it does not provide specific version numbers for these software components or any other libraries, which are necessary for reproducible dependency information. |
| Experiment Setup | Yes | We evaluated the fuzzing heuristics for 3 sizes of modification set: 1, 10, and 100. [...] For each heuristic, size, and recommender system (described next), we generated 100 modification sets. We then trained each recommender on the modified dataset and generate Top-10 recommendations for every user. [...] Funk SVD ... learns 25 latent features. [...] We experimented with several values of ϵ, and we chose 0.05 as the value that generally produced the most instability. |