# DAGs and all models are wrong motto, what's the implication?

Let's say I have a DAG and I find the right to way to estimate the causal effect of interest (which adjustment to make etc.). Then, I realize my model is wrong. Depending on how my model is wrong, my original assumptions may or may not be valid.

Since the common situation is (as the saying goes) the one in which all my models are wrong, is there any theorem about how sensitive to having a right model my causal identification strategy is?

To be (just) a bit more formal, let's say I have a models space M of possible models (DAGs). What is the probability that a model m does not correctly identify a causal effect? IS there any upper or lower bounds?

Let's say I can perform any experiment you want to estimate a causal effect with as many data points as you want. Is there a way for me to assess, from this new piece of evidence, if my original causal model had any chance of correctly identifying the causal effect?

I believe the language you are looking for is sensitivity analysis. Sensitivity analysis is the examination of the causal assumptions you made to identify your effect. Sensitivity analysis has been explored quite a bit over many years in the literature, going back quite a ways1. To answer your question, however, yes, you can put bounds on the causal effect, but the usefulness of those bounds may be very limited.

To provide an example I just want to reframe your question regarding the existence of multiple DAGs. Instead, I would look at it as a problem that some unobserved variable could exist that would break the backdoor path criterion in the DAG you have developed.

A simple example is that you have a direct path from treatment to outcome, $$X \rightarrow Y$$, and a confounder that affects both, $$Z$$. The concern would be that another, unobserved confounder, $$W$$, exists that would bias your results.

We know that for the confounder to bias our estimate it consists of two things: the imbalance between treated and control and the magnitude of the confounding relationship. Therefore, if we assume some level of imbalance and use prior knowledge to estimate bounds on the magnitude and it's the direction, we could estimate the bias induced by an unobserved confounder.

The first article I cite below is used as an example of sensitivity analysis. If you want to get into this more check out the second citation. In the introduction the cite several prior articles. Also, the methodology is really interesting.

• Thanks for the references, they seem to be a good way to address my concerns. Feb 26, 2019 at 18:27

To complement Landon's answer, let me elaborate a bit further.

Causal inference always requires untestable assumptions, the usual ones are absence of direct effects among variables (exclusion restrictions) or absence of unobserved common causes among variables (independence restrictions). For now, let us focus on the violation of these two, but, of course, there are other built-in assumptions as well, such as no selection bias, correctly measured variables, no interference between units and so on.

So the first thing I would point out is that DAGs, as models, have no special status under the "all models are wrong" motto---if you don't write down the implied DAG of your model, your model will still be "wrong" (or, better, "not useful"). Wherr DAGs can really help is to make it easier for you (and your peers) to see where your model could be wrong, and to better pin down where the sources of disagreements are. Then you can assess whether your conclusions are sensitive to that disagreement.

To perform this task, we need tools to derive sensitivity curves of the target quantity of interest in our causal models. Regarding linear structural models, we have just started developing algorithms for making this type of sensitivity analysis automatic and systematic. For instance, take the following example: Suppose you posit model $$G_O$$ and obtain that the causal effect of $$X$$ on $$Y$$ is identified (and given by the regression coefficient adjusting for $$Z$$). Now someone challenges you and say that your assumption of no unobserved confounders between $$Z$$ and $$X$$ is unreasonable, which leads to the alternative model model $$G_A$$. In $$G_A$$, however, the causal effect is not identifiable anymore. So what can you do? Here is where sensitivity analysis comes in.

Instead of point identifying the causal effect, you are going to express the causal effect as a function of other unidentifiable parameters of the model---such as the strength of the unobserved confounders. Then you can see how sensitive your conclusions are to different strengths of that parameter, and resort to outside knowledge and scientific plausibility judgments on those parameters to bound the causal effect of interest.

So the first task we need to solve is to decide whether information on some parameters, say, the strength of confounding between $$X$$ and $$Z$$, is sufficient for the identification of the quantity of interest and to find the correct estimand. Once you do that, you can use it to see how sensitivity your estimate is to violations of the zero confounding assumption.

So, back to the example, in $$G_A$$, can you use that information to bound the causal effect of $$X$$ on $$Y$$? The answer here is yes, and we can algorithmically derive the sensitivity curve (if the model were $$G_B$$, the answer would be no). But suppose you don't have direct external information about the confounders themselves, but you do have some prior studies that give plausible bounds on the causal effect of $$Z$$ on $$X$$. Can we use that information for the sensitivity analysis of $$X$$ and $$Y$$ instead? Here the answer is also yes. In this way we are building tools to have a disciplined discussion about violations of assumptions on arbitrary causal models (as represented by DAGs).