Privacy Preserving Adaptive Experiment Design
Authors: Jiachun Li, Kaining Shi, David Simchi-Levi
ICML 2024 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | In this section, we use numerical results to illustrate the theoretical findings and performance of the proposed algorithms. In particular, we focus on evaluating the algorithm Con SE in 2. First of all, we want to mention that Con SE is in fact a meta algorithm consisting of two phases, one for regret minimization and one for statistical estimation. For each phase, one can adopt scenario-specific algorithms to enhance potential better performance. In this paper, we adopt the batched sequential elimination for regret minimization, and RCT for CATE estimation, mainly because it s easier to privatize these two algorithms. However, when privacy is not a primary concern, one may adopt the celebrated UCB or Thopmson Sampling for regret minimization, and other estimation algorithms like adaptive neyman allocation proposed in (Zhao, 2023), or double machine learning methods with further structure assumption to improve estimation efficiency. In particular, when the outcome is assumed to have a linear structure, it was shown in (Kim et al., 2021) that doubly robust method can be adopted to capture the information in missing data. Below, we provide numerical results for estimation accuracy for both our RCT estimation and double machine learning estimation under different regret budget. We denote the mean difference estimator as MD, and double machine learning estimator as DML. The length of every experiment is 20000. For simplicity, we don t consider the heterogeneity of treatment effect among different features and assume no existence of features. Exp 1: Normal Linear Bandit In experiment 1, we set a0 = [1, 0], a1 = [0, 1], θ = [1, 1], µ1 = µ0 = 1. The experiment results are averaged on 50 replications. From the experiment results, we can conclude that: 1. DML for predicting missing data is not helpful when experiment length n is sufficiently large. Table 2. Normal Linear Bandit MD Error DML Error Reg=200 0.31 0.53 Reg=400 0.03 0.07 Reg=800 0.008 0.02 Exp 2: When regularity assumption of Linear Bandit Fails In experiment 2, we set a0 = [1, 0.001], a1 = [1, 0], µ0 = 3, µ1 = 1. In this case, the linear structure is ill-conditioned. The experiment results are averaged on 50 replications. In this experiment, MD estimator significantly outperforms DML estimator, which shows that 3. DML estimator is very sensitive in extreme cases, while MD estimator is robust. 4. MD estimator is much more efficient. The experiment of MD estimator can be finished within 1 second, while it takes 5 minutes to finish experiment of DML estimator. So to conclude, our numerical results validate the regretestimation tradeoff and show that naive MD estimator is already efficient and powerful when experiment is long enough. |
| Researcher Affiliation | Academia | Jiachun Li * 1 Kaining Shi * 2 David Simchi-Levi * 1 1Laboratory for Information and Decision Systems, MIT, Cambridge, U.S. 2Department of Statistics, The University of Chicago, Chicago, U.S.. Correspondence to: Jiachun Li <jiach334@mit.edu>. |
| Pseudocode | Yes | Algorithm 1 Con SE 1: Input: α 2: Initialize: Sj {0, 1}, epoch ej 0, rj 0, µj i 0, nj 0 (i = 0, 1; j = 1, 2, ..., M) 3: for t = 1 to [ n 2 ] do 4: Get feature xt = Xjt X 5: Increment njt njt + 1 6: if |Sjt| = 2 then 7: Select action at {0, 1} with equal probabilities ( 1 2) and update mean µjt at 8: Increment rjt rjt + 1 9: if rjt Rejt then 10: if ejt 1 then 11: Remove arm i from Sjt if max{ µjt 1 , µjt 2 } µjt i > 2he (i = 0, 1) 12: end if 13: Increment epoch ejt ejt + 1 14: Set rjt 0 15: Zero means: µjt i 0 i {1, 2} 16: end if 17: else 18: Pull the arm in Sjt 19: end if 20: if t = [ n 21: ˆfj = nj(1 j M) 22: Tmin = max{log n, min{ ˆf 1 α 1 , ˆf 1 α 2 , , ˆf 1 α M }} 23: end if 24: end for 25: for j = 1 to M do 26: nj = 0 27: end for 28: for t = [ n 2 ] + 1 to n do 29: Get feature xt = Xjt X 30: Increment njt njt + 1 31: if njt Tmin then 32: Select action at {0, 1} with equal probabilities ( 1 2) and update mean µjt at 33: if njt = Tmin then 34: Output ˆ (Xjt) = µjt 1 µjt 0 35: end if 36: else 37: Pull the arm in Sjt. (if |Sjt| = 2, pull any arm at Sjt) 38: end if 39: end for |
| Open Source Code | No | The paper does not include any statement about releasing open-source code or provide links to a code repository for the described methodology. |
| Open Datasets | No | The paper describes a |
| Dataset Splits | No | The paper describes numerical experiments but does not explicitly provide information on training, validation, or test dataset splits, such as percentages or sample counts. |
| Hardware Specification | No | The paper does not provide any specific details about the hardware used to run the experiments, such as GPU models, CPU types, or cloud computing instance specifications. |
| Software Dependencies | No | The paper mentions |
| Experiment Setup | Yes | The length of every experiment is 20000. For simplicity, we don t consider the heterogeneity of treatment effect among different features and assume no existence of features. Exp 1: Normal Linear Bandit In experiment 1, we set a0 = [1, 0], a1 = [0, 1], θ = [1, 1], µ1 = µ0 = 1. The experiment results are averaged on 50 replications. Exp 2: When regularity assumption of Linear Bandit Fails In experiment 2, we set a0 = [1, 0.001], a1 = [1, 0], µ0 = 3, µ1 = 1. In this case, the linear structure is ill-conditioned. The experiment results are averaged on 50 replications. |