Local Intrinsic Dimensional Entropy
Authors: Rohan Ghosh, Mehul Motani
AAAI 2023 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Experiments and Discussions Here we showcase two experiments, where we compute the ID-Entropy of the feature layers of classifier and autoencoder architectures on MNIST and CIFAR-10, and contrast it with generalization performance. |
| Researcher Affiliation | Academia | Rohan Ghosh1, Mehul Motani1,2 1 College of Design and Engineering, Department of Electrical and Computer Engineering, National University of Singapore 2 N.1 Institute for Health, Institute for Digital Medicine (Wis DM), Institute of Data Science, National University of Singapore |
| Pseudocode | Yes | Algorithm 1: Estimation of ID-Entropy Input: S = {X1, .., Xm} (i.i.d samples of RV X), a global ID estimator f ID(S ) of points in S , & parameters (k, n). Output: IDX (Estimate of ID-Entropy of X) 1: IDsum = 0 2: for j = 1, 2m/n , 3m/n , .. . . . , m do 3: Let S = k-nearest neighbors of Xj in S 4: IDsum = IDsum + f ID(S ) 5: IDX = IDsum/n |
| Open Source Code | Yes | code is available at: https://github.com/kentridgeai/ID-Entropy. |
| Open Datasets | Yes | Experiments and Discussions Here we showcase two experiments, where we compute the ID-Entropy of the feature layers of classifier and autoencoder architectures on MNIST and CIFAR-10 |
| Dataset Splits | No | The paper mentions 'training data size' and 'test accuracy' for MNIST and CIFAR-10 but does not specify explicit training, validation, or test dataset splits (e.g., percentages or sample counts). |
| Hardware Specification | No | The paper does not provide specific hardware details such as GPU models, CPU types, or memory specifications used for running the experiments. |
| Software Dependencies | No | The paper does not provide specific software dependencies with version numbers (e.g., Python, TensorFlow, PyTorch versions). |
| Experiment Setup | Yes | For all experiments we estimated the ID-entropy using Algorithm 1, with n = 2000 and k = 100. We used Fisher’s intrinsic dimensionality estimator in (Bac and Zinovyev 2020) as the global ID estimator f ID, as we found it to be a robust choice. With a fixed 4-layer CNN architecture for MNIST and a Res Net-44 for CIFAR-10, we repeat the training routine with different choices of the training data size and random network initializations. For both MNIST (4-layer CNN) and CIFAR-10 (Res Net-44), we add label noise by randomly changing the label of a training data point with a certain probability. |