Simple initialization and parametrization of sinusoidal networks via their kernel bandwidth
Authors: Filipe de Avila Belbute-Peres, J Zico Kolter
ICLR 2023 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We show that these simple sinusoidal networks can match and outperform SIRENs in implicit representation learning tasks, such as fitting videos, images and audio signals. We confirm, through an empirical analysis this theoretically predicted behavior also holds approximately in practice. |
| Researcher Affiliation | Collaboration | Filipe de Avila Belbute-Peres Carnegie Mellon University Pittsburgh, PA filiped@cs.cmu.edu J. Zico Kolter Carnegie Mellon University & Bosch Center for AI Pittsburgh, PA zkolter@cs.cmu.edu |
| Pseudocode | No | No pseudocode or clearly labeled algorithm blocks were found in the paper. |
| Open Source Code | No | The paper does not provide any specific links or explicit statements about the availability of open-source code for the methodology described. |
| Open Datasets | Yes | In the image fitting experiment, we treat an image as a function from the spatial domain to color values (x, y) (r, g, b). In the case of a monochromatic image, used here, this function maps instead to one-dimensional intensity values. We try to learn a function f : R2 R, parametrized as a sinusoidal network, in order to fit such an image. The input image used is 512 512, mapped to an input domain [ 1, 1]2. Both audios use a sampling rate of 44100Hz. The Bach audio is 7s long and the counting audio is approximately 12s long. The training set is created by randomly sampling 2,000 points from the available exact solution grid (shown in Figure 5). |
| Dataset Splits | Yes | To do this, in all experiments in this section we segment the input signal into training and test sets using a checkerboard pattern along all axis-aligned directions, points alternate between belonging to train and test set. |
| Hardware Specification | No | The paper does not provide specific hardware details (e.g., GPU/CPU models, memory specifications) used for running its experiments. |
| Software Dependencies | No | The paper mentions software components like 'Adam optimizer' and 'L-BFGS' but does not specify their version numbers. |
| Experiment Setup | Yes | The sinusoidal network used is a 5-layer MLP with hidden size 256, following the proposed initialization scheme above. The parameter ω is set to 32. The Adam optimizer is used with a learning rate of 3 10 3, trained for 10,000 steps in the short duration training results and for 20,000 steps in the long duration training results. For the gradient experiments, in short and long training results, a learning rate of 1 10 4 is used, trained for 10,000 and 20,000 steps respectively. For the Laplace experiments, in short and long training results, a learning rate of 1 10 3 is used, trained for 10,000 and 20,000 steps respectively. For both experiments, the parameter ω is set to 32 and the Adam optimizer is used. |