Sample-based approximate regularization
Authors: Philip Bachman, Amir-Massoud Farahmand, Doina Precup
ICML 2014 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We also examine the empirical performance of SAR on several datasets. In the rest of this paper, we present our generalization of smoothness-inducing, derivative-based regularizers (Section 2), present our approach for approximating them efficiently (Section 3), analyze its theoretical properties (Section 4), and present empirical results illustrating the power of the proposed approach (Section 5). |
| Researcher Affiliation | Academia | 1School of Computer Science, Mc Gill University, Montreal, Canada 2Carnegie Mellon University, Pittsburgh, USA |
| Pseudocode | Yes | Algorithm 1 SAR( px, psx, N, φ, i, ϵ ) Algorithm 2 Fuzzy Point Sampler( Dn, N, L ) Algorithm 3 Blurry Box Sampler( Dn, N, L ) |
| Open Source Code | No | The paper does not provide any concrete access information (e.g., repository link, explicit statement of code release) for open-source code. |
| Open Datasets | Yes | Our first tests involved least-squares regression with inputs x R and outputs y R. Using Synth Sin1d, we compared the performance of SAR2 regularization with L2 regularization and RKHS regularization of Gaussian RBFs. We tested SAR with the Boston housing dataset from UCI/Stat Lib, which comprises 506 observations x R13. Our final tests used the standard USPS/MNIST digit recognition datasets. |
| Dataset Splits | Yes | We performed 100 tests at each of the same training sizes as for Synth Sin1d. For each training size, 100 training sets were sampled from Synth Sin1d (with output noise) and, for each set, the function learned with each regularizer was tested on 5000 points sampled from Synth Sin1d (without output noise). We performed tests with training sets of size 150-450. For each size, 100 rounds of randomized cross validation were performed, with non-training examples used for evaluating performance. |
| Hardware Specification | No | The paper does not provide any specific hardware details (e.g., GPU/CPU models, memory amounts) used for running the experiments. |
| Software Dependencies | No | The paper mentions that the methods were "implemented in C++" but does not provide specific version numbers for compilers or any libraries used, which are required for a reproducible description of software dependencies. |
| Experiment Setup | Yes | Regularization weights for each method were set independently for each training size, to maximize measured performance. When boosting trees, we set the maximum depth to 3 and performed 250 rounds of boosting with a shrinkage factor of 0.1, which maximized measured performance. L2, SAR4, and Gauss all used full Gaussian RBFs centered on each training point with bandwidth γ = 0.05 fixed across all tests. P-spline applied 4th-order regularization to 2nd-order additive B-spline bases with 30 knots per dimension. SAR4 used a compound point sampler which drew 75% of its samples from the fuzzy point sampler in Algorithm 2 and 25% of its samples from the blurry box sampler in Algorithm 3. An unbiased direction sampler with stochastic lengths was used. The length distributions L in point/direction sampling were set to the non-negative half of a normal distribution, with standard deviation set to 0.5/0.2 times the median nearest-neighbor distance in the training set. A lower bound of 0.05 was set on the effective step length ϵ. We optimized the 1-vs-all squared hinge loss. |