When is it inappropriate to control for a variable?

Question

I can think of at least one naive example. Suppose I want to study the relationship between X and Z. I also suspect that Y influences Z, so I control for Y. However, as it turns out, unbeknownst to me, X causes Y, and Y causes Z. Therefore, by controlling for Y, I "cover up" the relationship between X and Z, since X is independent of Z given Y.

Now, in the previous example, it may be the case that the relationships I should be studying are the ones between X and Y, and Y and Z. However, if I knew such things a priori, I wouldn't be doing science in the first place. The study that I DID do now suggests that there is no relationship between X and Z, which is not the case.... X and Z ARE related.

This is illustrated in the following dependence-diagram. In the right scenario, Z depends on X and Y and X and Y are independent. We rightly control for Y to determine the relationship between X and Z. In the left scenario Z depends on Y which depends on X. X and Z are independent given Y, so the relationship between X and Z is "covered up" by controlling for Y.

My question is basically "When is it appropriate to control for variable Y and when not?"... It may be difficult or impossible to fully investigate the relationship between X and Y, but, for instance, controlling Y at a given level is an option. How do we decide before conducting our study, and what are common pitfalls of controlling too much or too little?

Citations appreciated.

Answer 1

Conditioning (i.e. adjusting) the probabilities of some outcome given some predictor on third variables is widely practiced, but as you rightly point out, may actually introduce bias into the resulting estimate as a representation of causal effects. This can even happen with "classical" definitions of a potential causal confounder, because both the confounder itself, and the predictor of interest may each have further causal confounders upstream. In the DAG below, for example, $L$ is a classic confounder of the causal effect of $E$ on $D$, because (1) it causes and is therefore associated with $E$, and (2) is associated with $D$ since it is associated with $U_{2}$ which is associated with $D$. However, either conditioning or stratifying $P(D|E)$ on $L$ (a "collider") will produce biased causal estimates of the effect of $E$ on $D$ because $L$ is confounded with $D$ by the unmeasured variable $U_{2}$, and $L$ is confounded with $E$ by the unmeasured variable $U_{1}$.

Understanding which variables to condition or stratify one's analysis on to provide an unbiased causal estimate requires careful consideration of the possible DAGs using the criteria for causal effect identifiability—no common causes that are not blocked by backdoor paths—described by Pearl, Robins, and others. There are no shortcuts. Learn common confounding patterns. Learn common selection bias patterns. Practice.

References

Greenland, S., Pearl, J., and Robins, J. M. (1999). Causal diagrams for epidemiologic research. Epidemiology, 10(1):37–48.

Hernán, M. A. and Robins, J. M. (2018). Causal Inference. Chapman & Hall/CRC, Boca Raton, FL

Maldonado, G. and Greenland, S. (2002). Estimating causal effects. International Journal of Epidemiology, 31(2):422–438.

Pearl, J. (2000). Causality: Models, Reasoning, and Inference. Cambridge University Press.

Answer 2

I believe the quick one-sentence answer to your question,

When is it appropriate to control for variable Y and when not?

is the "back-door criterion".

Judea Pearl's Structural Causal Model can tell you definitively which variables are sufficient (and when it's necessary) for conditioning, to infer the causal impact of one variable on another. Namely, this is answered using the back-door criterion, which is described in page 19 of this review paper by Pearl.

The major caveat is that it requires you to know the causal relationship between the variables (in the form of directional arrows in a graph). There's no way around that. This is where the difficulty and possible subjectivity can come into play. Pearl's structural causal model only allows you to know how to answer the right questions given a causal model (i.e. directed graph), which set of causal models are possible given a data distribution, or how to look for causal structure by performing the right experiment. It doesn't tell you how to find the right causal structure given only the data distribution. In fact, it claims that this is impossible without using external knowledge/intuition about the meaning of the variables.

The back-door criteria can be stated as follows:

To find the causal impact of $X$ on $Y,$ a set of variable nodes $S$ is sufficient to be conditioned on as long as it satisfies both of the following criteria:

1) No elements in $S$ is a descendant of $X$

2) $S$ blocks all "back-door" paths between $X$ and $Y$

Here, a "back-door" path is simply a path of arrows that begin at $Y$ and end with an arrow pointing at $X.$ (The direction that all other arrows point is not important.) And "blocking" is, itself, a criterion that has a specific meaning, which is given in page 11 of the above link. This is the same criterion that you would read when learning about "D-separation". I personally found that chapter 8 of Bishop's Pattern Recognition and Machine Learning describes the concept of blocking in D-separation far better than the Pearl source I linked above. But it goes like this:

A set of nodes, $S,$ blocks a path between $X$ and $Y$ if it satisfies at least one of the following criteria:

1) One of the nodes in the path, that is also in $S,$ emits at least one arrow on the path (i.e. the arrow is pointing away from the node)

2) A node that is neither in $S$ nor an ancestor of a node in $S$ has two arrows in the path "colliding" towards it (i.e. meeting it head-to-head)

This is an or criterion, unlike the general back-door criterion which is an and criterion.

To be clear about the back-door criterion, what it tells you is that, for a given causal model, when conditioning on a sufficient variable, you can learn the causal impact from the probability distribution of the data. (As we know, the joint distribution alone isn't sufficient for finding causal behavior because multiple causal structures can be responsible for the same distribution. This is why the causal model is required as well.) The distribution can be estimated using ordinary statistical/machine learning methods on the observational data. So as long as you know that the causal structure allows for conditioning on a variable (or set of variables), your estimate of the causal impact of one variable on another is as good as your estimate of the distribution of the data, which you obtain through statistical methods.

Here is what we find when we apply the back-door criterion to your two diagrams:

In neither case does there exist a back-door path from $Z$ to $X.$ So it's true that $Y$ blocks "all" back-door paths, because there aren't any. However, in the left diagram, $Y$ is a direct descendant of $X,$ while in the right diagram it is not. Therefore $Y$ follows the back-door criterion in the right diagram, but not the left. These are unsurprising results.

What is surprising, however, is that in the right diagram, as long as it is the complete picture, you need not condition on $Y$ to get the full causal impact of $X$ on $Z$. (Said another way, the null set satisfies the back-door criteria, and is thus, sufficient for conditioning.) Intuitively this is true because the value of $X$ is not associated with that of $Y$ so for sufficient data you can simply average over the values of $Y$ to marginalize the effect of $Y$ on $Z.$ One objection to this point can be that the data is limited, so you don't have a representative distribution of $Y$ values. But recall that the back-door criterion assumes you have the probability distribution of the data. In that case you can analytically marginalize $Y.$ Marginalization over a finite data set is just an estimation. Also, note that it's highly unlikely that this is the complete picture. There are likely external factors that impact $X.$ If those factors are also associated with $Y$ in any way, then more work must be done to see if $Y$ must be conditioned on, or if it's even sufficient. If you draw another arrow pointing from $Y$ to $X$ then $Y$ becomes necessary to control.

Those are, of course, very simple examples where intuition is enough to know when $Y$ can or cannot be controlled for. But here's a couple more examples where it's not obvious by looking at the diagram, and you can use the back-door criteria. For the following diagram we ask if it is sufficient to control for $Y$ when determining the causal impact of $X$ on $Z.$

The first thing to note is that, in both cases, $Y$ is not a descendant of $X.$ So it passes that criterion. The next thing to note is that, in both cases, there are several backdoor paths from $Z$ to $X.$ Two in the left diagram and three in the right.

In the left diagram the backdoor paths are $Z \leftarrow Y \rightarrow X$ and $Z \leftarrow W \rightarrow B \leftarrow A \rightarrow X. \hspace{1mm}$ $Y$ blocks the first path because it is an arrow-emitting node that is directly in the path. $Y$ also blocks the second path because it is neither $B,$ nor is it a descendant of $B,$ which is the only arrow colliding node in the path. Therefore $Y$ is a sufficient set for conditioning. (Note, unlike in your right diagram, the null set is not sufficient for conditioning, because it does not block the path $Z \leftarrow Y \rightarrow X$.)

In the right diagram the backdoor paths are the same two as in the left, plus the path $Z \leftarrow W \rightarrow B \rightarrow Y \rightarrow X. \hspace{1mm}$ $Y$ does block this path, because it is an arrow emitting node in the path. It also blocks the path $Z \leftarrow Y \rightarrow X$ for the same reason as the left diagram. However, it does not block the path $Z \leftarrow W \rightarrow B \leftarrow A \rightarrow X,$ because it is a direct descendant of the collider node $B.$ Therefore it is not sufficient for conditioning.

It is pretty unintuitive to see why $Y$ is sufficient for conditioning on the left diagram, because of the exogenous variables $A$ and $W$ that affect $X$ and $Z$ respectively. However, suppose there was no $B.$ In that case, there would be no spurious relationship between $X$ and $Z$ due to these exogenous variables so they're not of concern. The existence of $B,$ however, puts that in question. If $B$ is allowed to take whatever value it naturally takes given $A$ and $W$, it would not be a problem because it does not have any impact on the important variables, or the exogenous variables determining them. However, if $B$ (or any of its descendants) is controlled then it actually renders $A$ and $W$ dependent, which creates the spurious relationship beteween $X$ and $Z$ that we don't want. As mentioned in the linked source, this is an example of Berkson's paradox, where an observation of a variable caused by two independent sources makes those sources dependent (e.g. the result of two independent coin flips is rendered dependent upon the observation of the number of total heads flipped).

As I mentioned before the use of the back-door criterion requires that you know the causal model (i.e. the "correct" diagram of arrows between the variables). But the Structural Causal Model, in my opinion, also gives the best and most formal way to search for such a model, or to know when the search is futile. It also has the wonderful side effect of rendering terms like "confounding", "mediation", and "spurious" (all of which confuse me) obsolete. Just show me the picture and I'll tell you which circles ought to be controlled.

Answer 3

The following might or might not be appropriate to your case: if X is a treatment, then you might be able to go around your problem by using propensity score matching in which you would still keep the variable Y when you do the matching. In other words, you balance the covariates (Y is one of such covariates) that predict receiving the treatment X.
Note how there is no reference to the outcome variable Z in the above. You can also check how balanced your observations are (by generating a before and after matching balance table), which might give you insights into how much of X is determined by Y.