Accelerating Relative Entropy Coding with Space Partitioning
Authors: Jiajun He, Gergely Flamich, José Miguel Hernández-Lobato
NeurIPS 2024 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We conduct experiments on synthetic examples and neural codecs, including a VAE-based lossless codec on MNIST (Le Cun and Cortes, 1998) and INR-based lossy codecs on CIFAR-10 (Krizhevsky et al., 2009). |
| Researcher Affiliation | Academia | Jiajun He University of Cambridge jh2383@cam.ac.uk Gergely Flamich University of Cambridge gf332@cam.ac.uk José Miguel Hernández-Lobato University of Cambridge jmh233@cam.ac.uk |
| Pseudocode | Yes | Algorithm 1 Encoding of standard PFR Input: Q, P and a random state S. Output: Sample index N . # initialize: τ Ð 8, t0 Ð 0, N Ð 0 . rmax Ð supz ! qpzq ppzq ) . # run PFR: for n 1, 2, do Sample tn Expp1q; tn Ð tn 1 tn. # simulate a sample with PRNG: zn Ð PRNGp P, S, nq4 # update τ : τn Ð tn ppznq{qpznq. if τn ď τ then τ Ð τn, N Ð n. end if # check stopping criterion: if tn{rmax ą τ then break end if end for |
| Open Source Code | Yes | Specifically, we modify the block sizes and the REC algorithm in codes at https://github.com/cambridge-mlg/RECOMBINER (MIT License), and keep other settings unchanged. ... Codes for RECOMBINER6 (He et al., 2023): MIT License |
| Open Datasets | Yes | Lossless Compression on MNIST with VAE. As a further proof of concept, we apply our methods to losslessly compress MNIST images (Le Cun and Cortes, 1998). ... Lossy Compression on CIFAR-10 with INRs. We apply our methods to a more practical setting: compressing CIFAR-10 images with RECOMBINER (He et al., 2023), an implicit neural representation (INR)-based codec. ... Datasets: CIFAR-10 (Krizhevsky et al., 2009): MIT License MNIST (Le Cun and Cortes, 1998): CC BY-SA 3.0 License |
| Dataset Splits | No | We use Adam (Kingma and Ba, 2017) with a learning rate of 0.001 as the optimizer, to train the VAE with a batch size of 1,000 for 1,000 epochs. ... We estimate the distribution of t DKLr Q } Psu for each block from the training set, and entropy code it for each test image. ... The distribution of KL for each group, estimated from 60,000 training images. ... We also learn the prior ph using 15,000 images from the CIFAR-10 training set. ... on 100 CIFAR-10 test images |
| Hardware Specification | No | Our main contribution is the new REC algorithm, which can easily run on any CPU. We, in practice, run the scripts on many CPUs in parallel to get the error bar across multiple runs, but this is not needed. |
| Software Dependencies | No | We use Adam (Kingma and Ba, 2017) with a learning rate of 0.001 as the optimizer... To find the optimal value of ζ, we optimize the log-likelihood of the 50 indices with scipy.optimize.minimize. ... we fit a Gaussian Process regressor with scikit-learn package (Pedregosa et al., 2011)... encode the index using torchac package (Mentzer et al., 2019) |
| Experiment Setup | Yes | Specifically, both encoder and decoder are 2-layer MLP. We set the hidden size to 400 and the latent size to 100, with Re LU activation function. ... We use Adam (Kingma and Ba, 2017) with a learning rate of 0.001 as the optimizer, to train the VAE with a batch size of 1,000 for 1,000 epochs. ... We empirically find that our proposed method can handle a block size of DKLr Q } Ps 48 bits while maintaining DKLr Q } P 1s within a manageable range, approximately 12-14 bits. ... we opt to use 216 samples for each block. |