Manifold Topology Divergence: a Framework for Comparing Data Manifolds.
Authors: Serguei Barannikov, Ilya Trofimov, Grigorii Sotnikov, Ekaterina Trimbach, Alexander Korotin, Alexander Filippov, Evgeny Burnaev
NeurIPS 2021 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We apply it to assess the performance of deep generative models in various domains: images, 3D-shapes, time-series, and on different datasets: MNIST, Fashion MNIST, SVHN, CIFAR10, FFHQ, chest X-ray images, market stock data, Shape Net. We demonstrate that the MTop-Divergence accurately detects various degrees of mode-dropping, intramode collapse, mode invention, and image disturbance. Our algorithm scales well (essentially linearly) with the increase of the dimension of the ambient high-dimensional space. It is one of the first TDA-based practical methodologies that can be applied universally to datasets of different sizes and dimensions, including the ones on which the most recent GANs in the visual domain are trained. The proposed method is domain agnostic and does not rely on pre-trained networks. 3 Experiments We examine the ability of MTop-Div to measure quality of generative models trained on various datasets. |
| Researcher Affiliation | Collaboration | Serguei Barannikov Skolkovo Institute of Science and Technology Moscow, Russia CNRS, IMJ, Paris University, France Ilya Trofimov Skolkovo Institute of Science and Technology Moscow, Russia Grigorii Sotnikov Skolkovo Institute of Science and Technology Moscow, Russia Ekaterina Trimbach Moscow Institute of Physics and Technology Moscow, Russia Alexander Korotin Skolkovo Institute of Science and Technology, Artificial Intelligence Research Institute (AIRI), Moscow, Russia Alexander Filippov Huawei Noah s Ark Lab Evgeny Burnaev Skolkovo Institute of Science and Technology, Artificial Intelligence Research Institute (AIRI), Moscow, Russia |
| Pseudocode | Yes | Algorithm 1 Cross-Barcodei(P, Q) Input: m[P, P], m[P, Q] : matrices of pairwise distances within point cloud P, and between point clouds P and Q Require: VR(M): function computing filtered complex from pairwise distances matrix M Require: B(C, i): function computing persistence intervals of filtered complex C in dimension i b Q number of columns in matrix m[P, Q] m[Q, Q] zeroes(b Q, b Q) M m[P, P] m[P, Q] m[P, Q] m[Q, Q] Cross-Barcodei B(VR(M), i) Return: list of intervals Cross-Barcodei(P, Q) representing "births" and "deaths" of topological discrepancies Algorithm 2 MTop-Divergence(P, Q), see section 2.6 for details, default suggested values: b P = 1000, b Q = 10000, n = 100 Input: XP, XQ: NP D, NQ D arrays representing datasets for j = 1 to n do Pj random choice(XP,b P) Qj random choice(XQ,b Q) Bj list of intervals Cross Barcode1(Pj, Qj) calculated by Algorithm1 mtdj sum of lengths of all intervals in Bj end for MTop-Divergence(P, Q) mean(mtd) Return: number MTop-Divergence(P, Q) representing discrepancy between the distributions P, Q |
| Open Source Code | Yes | The source code is available at https://github.com/Ilya Trofimov/MTop Div. |
| Open Datasets | Yes | We apply it to assess the performance of deep generative models in various domains: images, 3D-shapes, time-series, and on different datasets: MNIST, Fashion MNIST, SVHN, CIFAR10, FFHQ, chest X-ray images, market stock data, Shape Net. The source code is available at https://github.com/Ilya Trofimov/MTop Div. We used the ACGAN implementation https://github.com/clvrai/ACGAN-PyTorch, (MIT License) and chest X-ray data was from https://www.kaggle.com/tawsifurrahman/covid19-radiography-database (Kaggle Data License) We evaluated the performance of Style GAN [17] and Style GAN2 [18] generators trained on the FFHQ dataset3. 3https://github.com/NVlabs/ffhq-dataset (CC-BY 2.0 License) |
| Dataset Splits | No | The paper mentions using a 'test set' and a 'subsample of the train set' for comparisons but does not explicitly detail training, validation, and test splits with specific percentages or sample counts, nor does it refer to predefined validation splits. |
| Hardware Specification | Yes | For example, for D = 3.15 · 106, and batch sizes b P = 103, b Q = 104, on NVIDIA TITAN RTX the time for GPU accelerated calculation of pairwise distances was 15 seconds, and GPU-accelerated calculation of Cross-Barcode1(P, Q) took 30 seconds. |
| Software Dependencies | No | The paper mentions using PyTorch in the context of a third-party ACGAN implementation, but it does not specify version numbers for any software dependencies used in their experiments (e.g., Python, libraries, frameworks). |
| Experiment Setup | Yes | Algorithm 2 MTop-Divergence(P, Q), see section 2.6 for details, default suggested values: b P = 1000, b Q = 10000, n = 100. We generated 20 · 103 samples with two truncation levels: ψ = 0.7, 1.0 and compared them with 20 · 103 samples from FFHQ. We trained an ACGAN [26] on a dataset consisting of chest X-rays of COVID-19 positive and healthy patients. Next, we studied the training process of ACGAN. Every 10 th epoch we evaluated the performance of ACGAN by comparing real and generated COVID-19 positive chest X-ray images. We trained GAN for 1000 epochs and tracked the following standard quality measures: Minimum Matching Distance (MMD), Coverage, and Jensen-Shannon Divergence (JSD) between marginal distributions. |