Interventionally Consistent Surrogates for Complex Simulation Models
Authors: Joel Dyer, Nicholas Bishop, Yorgos Felekis, Fabio Massimo Zennaro, Anisoara Calinescu, Theodoros Damoulas, Michael Wooldridge
NeurIPS 2024 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We further demonstrate with empirical studies that conventionally trained surrogates can misjudge the effect of interventions and misguide decision-makers towards suboptimal interventions, while surrogates trained for interventional consistency with our method closely mimic the behaviour of the original simulator under interventions of interest. Here, we outline a case study in which we learn interventionally consistent surrogates for the spatial SIRS ABM from Example 1, allowing us to experiment more rapidly with policy interventions while remaining confident that the causal behaviour of the original SIRS ABM is approximately preserved. |
| Researcher Affiliation | Academia | Joel Dyer University of Oxford Nicholas Bishop University of Oxford Yorgos Felekis University of Warwick Fabio Massimo Zennaro University of Bergen Anisoara Calinescu University of Oxford Theodoros Damoulas University of Warwick Michael Wooldridge University of Oxford |
| Pseudocode | Yes | Algorithm 1: Summary of the training procedure. Input: Budget R; batch size B J1, R 1K; ABM M; intervention distribution η; surrogate family MΨ; abstraction map family ΩΦ Result: Trained surrogate and abstraction map parameters, ψ and ϕ Set D = ; for r = 1 to R do Sample ι(r) η, x(r) PMι(r); D D (ι(r), τ(x(r))) end while not converged do Sample minibatch {(ι(b), τ(x(b)))}B b=1 uniformly from D; Take gradient step in ϕ, ψ using Equation 9 end |
| Open Source Code | Yes | Code for reproducing the experimental results is available at https://github.com/joelnmdyer/neurips_ics4csm. |
| Open Datasets | No | A total number of R = 1000 training samples was generated from the ABM for each of the observational and interventional training sets; these were each split 5 times into different training and validation sets of sizes 800 and 200, respectively, with a new surrogate model trained from scratch on each of these splits. The paper generates its own data from a simulation model rather than using a pre-existing public dataset. |
| Dataset Splits | Yes | A total number of R = 1000 training samples was generated from the ABM for each of the observational and interventional training sets; these were each split 5 times into different training and validation sets of sizes 800 and 200, respectively, with a new surrogate model trained from scratch on each of these splits. |
| Hardware Specification | Yes | All models were trained on CPU on a 2022 Mac Book Pro, operating on mac OS Ventura 13.2.1. |
| Software Dependencies | No | Software dependencies are specified in the Git Hub repository containing the code for this paper, which will be made public upon acceptance. The paper states that software dependencies are specified in the GitHub repository, but does not list them directly within the paper or its appendices. |
| Experiment Setup | Yes | For all surrogates, the neural networks comprising the ωϕ map and structural equations parameterised by ψ were trained with a learning rate of 10-2 for a maximum number of 1000 epochs, batch size B = 50, and with the Adam optimiser [Kingma and Ba, 2014]. A total number of R = 1000 training samples was generated from the ABM for each of the observational and interventional training sets; these were each split 5 times into different training and validation sets of sizes 800 and 200, respectively, with a new surrogate model trained from scratch on each of these splits. We apply an early stopping criterion in which training is ceased if the validation error does not decrease for 20 consecutive epochs. |