Online Control of the False Coverage Rate and False Sign Rate
Authors: Asaf Weinstein, Aaditya Ramdas
ICML 2020 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | 5. Simulation We carry out experiments where online confidence intervals are constructed under different (predictable) selection schemes. Setting α = 0.1, in each of N = 10, 000 simulations we draw m = 10, 000 parameters i.i.d. from a mixture ( 0.5δ10 3 + 0.5δ 10 3, w.p. 0.9 1 + Wi, w.p. 0.1 , where Wi Pois(1). The mass at 10 3 represents the null component (essentially zero), while the nonnulls Online False Coverage Rate Control are drawn so that large effects are rare. We then draw the observations Xi N(θi, 1). In our implementation, the LORD-CI algorithm always uses the sequence of αi of the LORD++ procedure (Ramdas et al., 2017) with W0 = α/2, γ(j) = 0.0722 log(j 2) je log j , as in the experiments of Javanmard & Montanari (2018); Ramdas et al. (2017). For a conditional CI we used the construction from Weinstein et al. (2013, Section 2) obtained by inverting shortest acceptance regions. |
| Researcher Affiliation | Academia | 1School of Computer Science and Engineering, Hebrew University of Jerusalem 2Departments of Statistics & Data Science, and Machine Learning, Carnegie Mellon University. |
| Pseudocode | Yes | Algorithm 1 A monotone instantiation of LORD-CI Input :sequence {Xi} observed sequentially; prespecified deterministic sequence {γi} summing to one; constant W0 (0, α); arbitrary selection rules {Si}; marginal CI rules {Ii}. Output:online FCR-adjusted selective CIs t 1 // tracks time for j = 1, 2, ... do while Si(Xi) = 0 do t t + 1 // θt is not selected, increment time end τj t // time of the j-th selection αt γt W0 + (α W0)γt τ1 + α X {k:τk<t,τk =τ1} γt τk Report It = It(Xt, αt) t t + 1 end |
| Open Source Code | No | The paper does not provide any statement or link regarding the availability of its source code. |
| Open Datasets | No | The paper describes a synthetic data generation process for its simulations, specifying how parameters and observations are drawn, but it does not use or provide access to a pre-existing public dataset. |
| Dataset Splits | No | The paper describes simulation settings but does not specify training, validation, or test dataset splits, as it generates synthetic data on the fly rather than using fixed datasets with predefined partitions. |
| Hardware Specification | No | The paper describes the simulation setup but does not provide any specific hardware details such as CPU/GPU models, memory, or type of computing environment used for running the experiments. |
| Software Dependencies | No | The paper mentions using specific algorithms and procedures (e.g., 'LORD++ procedure', 'construction from Weinstein et al. (2013, Section 2)') but does not list any specific software dependencies with version numbers (e.g., Python 3.8, PyTorch 1.9). |
| Experiment Setup | Yes | Setting α = 0.1, in each of N = 10, 000 simulations we draw m = 10, 000 parameters i.i.d. from a mixture ( 0.5δ10 3 + 0.5δ 10 3, w.p. 0.9 1 + Wi, w.p. 0.1 , where Wi Pois(1). The mass at 10 3 represents the null component (essentially zero), while the nonnulls Online False Coverage Rate Control are drawn so that large effects are rare. We then draw the observations Xi N(θi, 1). In our implementation, the LORD-CI algorithm always uses the sequence of αi of the LORD++ procedure (Ramdas et al., 2017) with W0 = α/2, γ(j) = 0.0722 log(j 2) je log j , as in the experiments of Javanmard & Montanari (2018); Ramdas et al. (2017). For a conditional CI we used the construction from Weinstein et al. (2013, Section 2) obtained by inverting shortest acceptance regions. |