Imagine that you are asked to infer some causal effect -- a change in an outcome $y$ in response to some variable $x$. But, the person asking for this directs you to use a predictive model to do so. Here's the setup:
- $x$ is confounded inasmuch as there is some unobserved $u$ that is causally linked to both $y$ and $x$. We have a classical omitted variables bias.
- We have high dimensional covariates $\mathbf{Z}$ that are not independent of $y$ or $x$ and/or $u$
- You are asked to train a suite of predictive models -- neural networks, boosted trees, whatever -- denoted $g_i([x, \mathbf{Z}]) + \epsilon$ where $i$ indexes different models, and then select among them model $i$ that minimizes some metric of predictive skill. RMSE, for instance.
- Based on the chosen model, you are asked to report $$ \frac{\partial \hat{y}}{\partial x} = \frac{\partial \hat{g}_i([x, \mathbf{Z}])}{\partial x} $$
- You know that $$ E\left[\frac{\partial \hat{y}}{\partial x}\right] \neq \frac{\partial y}{\partial x} $$ in the population, because the error term includes the omitted variable, so therefore $$ \frac{\partial \epsilon}{\partial x} \neq 0 \text{ in the population, despite the fact that } \frac{\partial \hat\epsilon}{\partial x} = 0 $$ in any reasonable model $g$.
On top of omitted variables bias, there may be bias from regularization too!
- Further assume that you have some causal model -- say an instrumental variables regression, utilizing some suitable instrument $w$ for $x$. It's one of the models in your suite of models, but its predictive skill in terms of cross-validated RMSE is worse than the others.
The best model is the one that produces the consistent causal estimate, right? But:
How would you explain this to someone in layperson's terms?
The person asking for analysis doesn't understand causal inference, and needs to be educated. However, they don't understand math and have little attention span. How can you effectively convey the basic point that causal methods are required, and predictive methods are inappropriate? No math, lots of stories, pithy sentences.