Metric Space Magnitude for Evaluating the Diversity of Latent Representations
Authors: Katharina Limbeck, Rayna Andreeva, Rik Sarkar, Bastian Rieck
NeurIPS 2024 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We show their utility and superior performance across different domains and tasks, including (i) the automated estimation of diversity, (ii) the detection of mode collapse, and (iii) the evaluation of generative models for text, image, and graph data. |
| Researcher Affiliation | Academia | Katharina Limbeck1,2 Rayna Andreeva3 Rik Sarkar3 Bastian Rieck1,2,4 1Helmholtz Munich 2Technical University of Munich 3University of Edinburgh 4University of Fribourg |
| Pseudocode | No | The paper does not contain structured pseudocode or algorithm blocks with explicit labels. |
| Open Source Code | Yes | The code for computing magnitude is available at https://github.com/aidos-lab/magnipy. |
| Open Datasets | Yes | We analyse data from Tevet and Berant [38]... To this end, we analyse 16384 documents of four different Hugging Face datasets... We randomly sample 100 images from each of the 10 classes in CIFAR10 and embed them... We analyse 3 synthetic (Lobster, Grid, and Community) and 2 real-world (Proteins and Ego) graph datasets |
| Dataset Splits | Yes | We analyse the performance of each diversity metric at predicting the ground-truth diversity scores, dec, using 5-fold cross-validation repeated 20 times, trained via isotonic regression models... Table 2 reports the results of 5-fold cross-validation with 20 repetitions for both prepossessing choices. |
| Hardware Specification | Yes | Indeed, we ran all our experiments locally with the following hardware specifications: CPU: 12th Gen Intel(R) Core(TM) i7-1265U, RAM: 16 GB DDR4, SSD: 512 GB NVMe SSD |
| Software Dependencies | No | The paper mentions software components like "sentence-transformer library" and "scipy.optimize.toms748", but does not provide specific version numbers for these or other key software dependencies. |
| Experiment Setup | Yes | For each text embedding we compute magnitude functions and our diversity measure, MAGAREA, across 20 evenly sampled scales in [0, tcut] where tcut is the median of the convergence scales across all embeddings setting ϵ = 0.01|X|... We compute magnitude for the corresponding graph embeddings across 40 evenly-spaced scales until the convergence scale of the reference choosing ϵ = 0.05|X|... We vary the number of layers between [2, 3, . . . , 7] and vary the hidden dimensions in the interval [5, 30] with an increment of 5 resulting in a total of 36 architectures. We repeat the experiments for 5 different random seeds. |