Relevance of overfitting while examining causal effects

Question

Let's assume that we have a well defined directed acrylic graph (DAG) showing correct causal relationships between variables.

And this DAG tells us that we need to adjust for 7 variables to analyse the direct effect of an exposure on an outcome.

We are not interested in finding the best prediction. We would like to know does the exposure cause changes in our outcome. Thus, should we worry about finding the best model, AIC values etc? Would it be enough to check that conditional effects are just reasonable numbers?

Answer 1

DAGs are non-parametric. If we assume that all variables in your problem are dichotomous (or categorical) and all assumptions required for valid causal inference are met, you can unbiasedly estimate the causal effect with a saturated model for the outcome adjusting for the 7 variables (alternatively, using inverse probability weighting, or both). Otherwise, and in common applications with limited data, you usually have to make some parametric assumptions to specify a parsimonious model, which can lead to bias due to model misspecification. Fit on observed data is not typically used to guide specification of structural models, although parsimonious models are often preferred for interpretability and reducing model variance.

When using machine learning methods for causal estimation, both regularization and overfitting can result in bias. Naive application of ML methods (i.e. that used for prediction) to estimate causal effects is generally invalid; they can not model individual-level causal effects out of the box. However, there are methods for semi-parametric estimation of causal effects where machine learning methods can be used to model high-dimensional confounders while still modelling the treatment effects parametrically for interpretability.

Answer 2

If you're not interested in prediction, the most important is to guarantee that you have the right model and all the relevant variables to te problem - which is different than "check that conditional effects are just reasonable numbers", you can't know from guess what would be a reasonable number. The overfitting problem (larger AIC) acts more on giving your estimators more variance, which isn't all that bad if you're not looking into prediction, as it won't change consistency of the estimators.

EDIT: As Richard pointed out, it's not as simple as I made it look like. Whether it's for prediction or explanation, it's always good to find parsimonious models. As i said, your main goal is to include every variable in the true model, but as you can't know for sure which those are, it might be good to not include the ones you don't have much justification in the name of smaller variance, because even if you have consistent estimators (and can talk about causality), large variance means that your estimates will be largely affected by noise on the sample.