Simulating Estimates with Potential Heterogenous Treatment Effects

Question

To verify whether a given model can accurately estimate the target estimand of interest, one might generate data to simulate the assumed data generating process, define a treatment effect, and estimate the target treatment effect and evaluate how well a given model recovers the estimand.

I am new to simulation, and I notice in other works that causal effects (whether these be treatment $\rightarrow$ outcome, confounder $\rightarrow$ outcome, confounder $\rightarrow$ treatment, etc.) are often specified in the aggregate. I.e., I might define and simulate my DGP as:

Outcome = Treatment(0.75) + Confounder$_1$(0.33) + Confounder$_2$(1.05) + ...

Of course, this definition does not account for any form of effect heterogeneity for either the treatment nor the confounder effects. Assume that I am not interested in estimating heterogenous treatment effects. I am simply interested in estimating the ATT for example, and I want to evaluate different models and see which recovers the target estimand the best. Do I need to worry about simulating potential heterogenous treatment effects? If I omit defining such possible effects from the simulation, is my simulated data inherently not reflective of the "real" data and therefore not informative as a validation check of different models?

Or, in contrast, if I am only interested in average effects, does the correct specification of sub-population-level effects not matter? That doesn't seem quite right to me, although, neither does the manual specification of any heterogenous treatment effect.

Answer 1

Do I need to worry about simulating potential heterogenous treatment effects? If I omit defining such possible effects from the simulation, is my simulated data inherently not reflective of the "real" data and therefore not informative as a validation check of different models?

For the latter part of this question, it depends on the goal of your simulation. There are (at least) three goals that a simulation study could have: a proof-of-concept for a new estimator, a comparison between existing estimators in a general setting, or a comparison between existing estimators in a particular application. The concern of the simulated data not reflecting the application is mostly a concern for the last goal.

The last goal also tends to be better addressed design-wise when one uses plasmode simulation. This simulates data from an existing data set. When implemented well, these simulation designs are likely to more closely approximate the particular application. For a paper on this general idea, see Athey et al.

To answer the first question, it depends on the goal of your simulation study. Selecting the question you want to address via simulation will help decide on how to specify the data generating mechanism.

Or, in contrast, if I am only interested in average effects, does the correct specification of sub-population-level effects not matter? That doesn't seem quite right to me, although, neither does the manual specification of any heterogenous treatment effect.

If you are interested in average effects, correct specification of the sub-population-level effects can still matter. For example, g-computation relies on the assumption that the outcome model is correctly specified. Therefore, it requires that the outcome model includes the corresponding interaction terms. However, inverse probability weighting does not because it is (generally) nonparametric for the outcome process. This distinction between estimators highlights why you might want to consider heterogeneous effects, even if interest is in only marginal parameters. Different estimators can rely on model different parts of the data generating process. In cases of homogeneous effects, these differences can be obscured.

The last statement is also unclear to me. It seems that the world has heterogeneous effects (any cause will be an effect measure modifier on at least one scale). Therefore, if our methods can't handle heterogeneous effects we should avoid using them. So, when studying the performance of estimators it is better to include heterogeneity.

Summary: Generally, I would include heterogeneity in outcome model in simulation studies, even for estimands like the average treatment effect. This practice makes it less likely to be led astray. However, if your goal is a initial proof-of-concept then not having them may be acceptable.