Targeted Sequential Indirect Experiment Design
Authors: Elisabeth Ailer, Niclas Dern, Jason S. Hartford, Niki Kilbertus
NeurIPS 2024 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | 4 Experiments Setup. We consider a low-dimensional setting (for visualization purpose) with dx = dz = 2 and... We use n = 250 samples in each round over a total of T = 16 experiments. ... We perform nseeds = 50 runs for different random seeds and report means with the 10 and 90-percentiles in Figure 1. |
| Researcher Affiliation | Collaboration | Elisabeth Ailer Technical University of Munich Helmholtz Munich Munich Center for Machine Learning (MCML) Niclas Dern Technical University of Munich Jason Hartford Valence Labs Niki Kilbertus Technical University of Munich Helmholtz Munich Munich Center for Machine Learning (MCML) |
| Pseudocode | Yes | G Algorithmic Boxes We present pseudocode for the explore then exploit (EE) strategy in Algorithm 1, for alternating explore exploit (AEE) in Algorithm 2, and for the adaptive strategy in Algorithm 3. |
| Open Source Code | Yes | The code to reproduce results is available at https://github.com/EAiler/targeted-iv-experiments. |
| Open Datasets | No | The paper uses a synthetic (simulated) dataset, not a pre-existing public dataset for which access information would typically be provided. The data generation process is described within the paper. |
| Dataset Splits | No | The paper describes collecting 'n = 250 samples in each round' and performing 'T = 16 experiments' and 'nseeds = 50 runs', but it does not specify explicit training, validation, or test dataset splits with percentages or sample counts. |
| Hardware Specification | Yes | All methods were run on a Mac Book Pro with Intel CPU. |
| Software Dependencies | No | Table 1 lists software dependencies like 'Python', 'Py Torch', 'Numpy', 'Pandas', 'Jupyter', 'Matplotlib', 'Scikit-learn', 'Sci Py', 'JAX', and 'SLURM' with references, but it does not provide specific version numbers for these software components, which is required for a reproducible description. |
| Experiment Setup | Yes | Method parameters. We use radial basis function (RBF) kernels k(x1, x2) = exp(ρ x1 x2 2) with a fixed ρ = 1. Note, that the three hyperparameters are relative weights. Thus we set λs := λgλf /λc = 0.01 and λc = 0.04, we refer to the Appendix C for a further comparison of different hyperparameter choices. For all strategies the variance of our policy is set at σe = 0.001. For explore then exploit, we chose T1 = 10, T2 = 6 with πt = N(µt, σe Iddz) and independent µt N(0dz, Iddz) for t [T1]. The Gaussian mixture of the adaptive strategy is initialized with M = 3, γ = (1/3, 1/3, 1/3), Σm = Iddz and independent µm N(0dz, Iddz). We use a constant learning rate αt = 0.01 for all t and restrict ourselves to learning only weights γm, means µm and the diagonal entries of the covariances Σm. |