Counterfactual Randomization: Rescuing Experimental Studies from Obscured Confounding
Authors: Andrew Forney, Elias Bareinboim2454-2461
AAAI 2019 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Our paper presents a novel experimental design that can be noninvasively layered atop past and future RCTs to not only expose the presence of UCs in a system, but also reveal patientand practitioner-specific treatment effects in order to improve decision-making. Applications are given to personalized medicine, second opinions in diagnosis, and employing offline results in online recommender systems. ... To validate the efficacy of the methodologies detailed in the previous sections, we first simulated an offline HI-RCT and then used the resulting data to inform an online HI-RDC agent in a HI-MABUC setting. |
| Researcher Affiliation | Academia | Andrew Forney Department of Computer Science Loyola Marymount University Los Angeles, CA Elias Bareinboim Department of Computer Science Purdue University West Lafayette, IN |
| Pseudocode | Yes | Algorithm 1 HI-RDC-RCT agent, parameterized by HI-RCT data D, number of samples in the calibration set n, and IEC clustering tolerance τ such that Thm. 4.2 allows for noisy correlation between IEC actor intents, ρ(IAi, IAj) 1 τ. |
| Open Source Code | Yes | Simulation source code available at: https://github.com/Forns/hi-mabuc-aaai19 |
| Open Datasets | No | The paper uses a simulated dataset for its experiments, which is generated by the simulation code. It does not provide access to a pre-existing publicly available or open dataset. |
| Dataset Splits | No | The paper describes generating a simulated dataset but does not provide explicit training, validation, or test dataset splits in terms of percentages or counts for a fixed dataset. It refers to a 'calibration unit set' but this is for learning IECs, not a standard dataset split for model evaluation. |
| Hardware Specification | No | The paper does not provide specific hardware details (like GPU/CPU models, processor types, or memory amounts) used for running its experiments. |
| Software Dependencies | No | The paper does not provide specific ancillary software details with version numbers (e.g., library or solver names with version numbers) needed to replicate the experiment. |
| Experiment Setup | Yes | The simulations consisted of N = 1000 Monte Carlo (MC) repetitions each composed of T = 10,000 units / trials. At each trial, the state of the UCs was sampled ut P(U), the intents of 10 actors belonging to the same two IECs as in the HI-RCT were instantiated i A t f IA(ut), the agent s policy π selects an arm xt, and an outcome yt to that choice is observed. After all MC repetitions were completed, the average, cumulative u-regret (Def. 4.6) was assessed as a metric for each agent s performance. ... all compared agents explore and exploit treatments using the Thompson Sampling procedure (Ortega and Braun 2014). ... calibrated with a set of size n = 20, but where datum in the calibration set were chosen at random from the simulated HI-RCT. ... agent described in Alg. 1 w/ calibration set size of n = 20. ... IEC clustering tolerance τ such that Thm. 4.2 allows for noisy correlation between IEC actor intents, ρ(IAi, IAj) 1 τ. |