Compression with Flows via Local Bits-Back Coding
Authors: Jonathan Ho, Evan Lohn, Pieter Abbeel
NeurIPS 2019 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We designed experiments to investigate the following: (1) how well local bits-back codelengths match the theoretical codelengths of modern flow models on high-dimensional data, (2) the effects of the precision and noise parameters δ and σ on codelengths (Section 3.3), and (3) the computational efficiency of local bits-back coding for use in practice. We focused on Flow++ [19], a recently proposed Real NVP-type flow with a flow-based dequantizer. We used all concepts presented in this paper: Algorithm 1 for elementwise and convolution flows [23], Algorithm 2 for coupling layers, the compositional method of Section 3.4.3, and Algorithm 3 for dequantization. We used asymmetric numeral systems (ANS) [12], following the BB-ANS [42] and Bit-Swap [26] algorithms for VAEs (though the ideas behind our algorithms do not depend on ANS). We expect our implementation to easily extend to other models, like flows for video [28] and audio [35], though we leave that for future work. We provide open-source code at https://github.com/hojonathanho/localbitsback. |
| Researcher Affiliation | Collaboration | Jonathan Ho UC Berkeley jonathanho@berkeley.edu Evan Lohn UC Berkeley evan.lohn@berkeley.edu Pieter Abbeel UC Berkeley, covariant.ai pabbeel@cs.berkeley.edu |
| Pseudocode | Yes | Algorithm 1 Local bits-back encoding: for black box flows (decoding in Appendix B) |
| Open Source Code | Yes | Open-source code is available at https://github.com/hojonathanho/localbitsback. |
| Open Datasets | Yes | Table 1 lists the local bits-back codelengths on the test sets of CIFAR10, 32x32 Image Net, and 64x64 Image Net. |
| Dataset Splits | No | The paper mentions using standard datasets like CIFAR10 and Image Net for testing but does not explicitly provide details on how the dataset was split into training, validation, and test sets or specific percentages/counts for each split. |
| Hardware Specification | Yes | We used Open MP-based CPU code for compression with parallel ANS streams [14], with neural net operations running on a GPU. See Table 2 for encoding timings (decoding timings in Appendix C are nearly identical), averaged over 5 runs, on 16 CPU cores and 1 Titan X GPU. |
| Software Dependencies | No | The paper mentions 'Open MP-based CPU code' and 'neural net operations running on a GPU' but does not provide specific version numbers for software dependencies such as Python, PyTorch, CUDA, or other relevant libraries. |
| Experiment Setup | Yes | Above, we fixed δx = δz = 2 32 and σ = 2 14, but here, we study the impact of varying δ = δx = δz and σ |