Second Order PAC-Bayesian Bounds for the Weighted Majority Vote
Authors: Andres Masegosa, Stephan Lorenzen, Christian Igel, Yevgeny Seldin
NeurIPS 2020 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We studied the empirical performance of the bounds using standard random forests [Breiman, 2001] on a subset of data sets from the UCI and Lib SVM repositories [Dua and Graff, 2019, Chang and Lin, 2011]. An overview of the data sets is given in Table I.1 in the appendix. The number of points varied from 3000 to 70000 with dimensions d < 1000. For each data set we set aside 20% of the data for a test set Stest and used the remaining data, which we call S, for ensemble construction and computation of the bounds. Forests with 100 trees were trained until leaves were pure, using the Gini criterion for splitting and considering d features in each split. We made 50 repetitions of each experiment and report the mean and standard deviation. In all our experiments π was uniform and δ = 0.05. We present two experiments: (1) a comparison of tightness of the bounds applied to uniform weighting, and (2) a comparison of weighting optimization the bounds. |
| Researcher Affiliation | Academia | Andrés R. Masegosa University of Almería andresma@ual.es Stephan S. Lorenzen Christian Igel Yevgeny Seldin University of Copenhagen {lorenzen,igel,seldin}@di.ku.dk |
| Pseudocode | No | The paper describes a "Bound Minimization procedure" in Appendix H but it is not presented in a structured pseudocode or algorithm block format. |
| Open Source Code | Yes | The python source code for replicating the experiments is available at Github2. 2https://github.com/Stephan Lorenzen/Majority Vote Bounds |
| Open Datasets | Yes | We studied the empirical performance of the bounds using standard random forests [Breiman, 2001] on a subset of data sets from the UCI and Lib SVM repositories [Dua and Graff, 2019, Chang and Lin, 2011]. |
| Dataset Splits | Yes | For each data set we set aside 20% of the data for a test set Stest and used the remaining data, which we call S, for ensemble construction and computation of the bounds. ... The idea is to generate multiple splits of a data set S into pairs of subsets S = Th Sh, such that Th Sh = . A hypothesis h is then trained on Th and ˆL(h, Sh) provides an unbiased estimate of its loss. ... the weighted empirical losses Eρ[ˆL(h, S)] in the bounds are replaced by weighted validation losses Eρ[ˆL(h, Sh)], and the sample size n is replaced by the minimal validation set size nmin = minh |Sh|. It is possible to use any data-independent prior, with uniform prior π(h) = 1/|H| being a natural choice in many cases [Thiemann et al., 2017]. For pairs of hypotheses (h, h ) we use the overlaps of their validation sets Sh Sh to calculate an unbiased estimate of their tandem loss, ˆL(h, h , Sh Sh ), which replaces ˆL(h, h , S) in the bounds. |
| Hardware Specification | No | The paper does not provide specific details about the hardware (e.g., GPU/CPU models, memory) used for running the experiments. |
| Software Dependencies | No | The paper mentions that "The python source code for replicating the experiments is available at Github" but does not specify software versions (e.g., Python version, specific library versions). |
| Experiment Setup | Yes | Forests with 100 trees were trained until leaves were pure, using the Gini criterion for splitting and considering d features in each split. We made 50 repetitions of each experiment and report the mean and standard deviation. In all our experiments π was uniform and δ = 0.05. ... Optimization details are provided in Appendix H. |