Is multicollinearity a "warning sign" for causal inference?

Question

Suppose we are inferring whether $A$ causes $B$, while holding $N = [N_0, N_1, \ldots, N_n]$ constant and we find $N_i$ correlates well but not perfectly with $A$. There are four reasons to exclude $N_i$:

1. We may have discovered a causal confound $N_i \rightarrow A$ (read: $N_i$ causes $A$) or $A \rightarrow N_i$. If there is a plausible causal relationship (either direction), then $N_i$ should not be controlled for, because it is a confounder ($N_i \rightarrow A$ and $N_i \rightarrow B$), a mediator ($A \rightarrow N_i \rightarrow B$).

2. The resulting coefficients will be unstable and potentially irreproducible; doing the experiment again might select a different pair of coefficients for $A$ and $N_i$ (I'm not assuming the correlation is perfect; just good enough to, with a sprinkling of statistical noise, select different coefficients).

3. Occam's Razor suggests a more parsimonious theory (that with $N_i$ removed) should be pursued first. If $A$ and $N_i$ are collinear, then a theory with just $A$ will be more parsimonious and almost as good as a theory with $A$ and $N_i$.

4. Having that additional coefficient (free parameter) also increases the chance of overfitting, since there are more degrees of freedom, which worsens AIC, BIC, and out-of-sample generalization.

So, why does Wikipedia claim multicollinearity among predictors is not a problem (multiple times in that article) and not a reason to exclude variables, stating

High collinearity indicates that it is exceptionally important to include all collinear variables, as excluding any will cause worse coefficient estimates, strong confounding, and downward-biased estimates of standard errors.

I agree that blindly removing them post hoc is unjustified, but it should warn the researcher to re-check their a priori reasoning for choosing those variables in the first place, assessing possibility of a mediating and confounding effect and whether the additional parameter is "worth it" in its predictive value.

Answer 1

Is multicollinearity a "warning sign" for causal inference?

It can be. I agree with you that there are plenty of legitimate concerns with multicollinearity, including the ones you raised. And they can be problematic for other kinds of modeling too, not just causal inference.

"why does Wikipedia claim multicollinearity among predictors is not a problem"?

Because whoever wrote those parts of the Wikipedia article was strongly opinionated :-) and not because it's an objective fact (it isn't) or everyone agrees (we don't). You are welcome to propose edits to Wikipedia yourself, relaxing some of their assertions and suggesting alternative strategies---including alternatives recommended by the articles already cited on Wikipedia, as you note in your comments.

That said, I imagine these wiki authors might be trying to push back against bad advice in the other direction. Too many outdated textbooks (and even new ones) implicitly or explicitly encourage a default process of "Just drop variable after variable until the remaining ones are significant," which leads to blatant p-hacking. Perhaps the wiki authors are tired of fighting this battle with students and colleagues, and may have overreacted in the other direction by recommending a default process of "Just keep all the variables in there no matter what."

Unfortunately, no short pithy advice paragraph on Wikipedia can replace deliberate, thoughtful training in this issues. The best strategy for handling multicollinearity in a given analysis depends on the context: what do you hope to learn, why did you collect these variables in the first place, how did you sample these units, what population are you studying, is this exploratory or confirmatory research, are you making predictions or inferences, will predictions be made on exactly the same population that this data represents, etc...?

Answer 2

1. This might be a good reason to exclude the variable. On the other hand, the regression that includes the variable still yields unbiased and consistent estimates, so, assuming otherwise correct specification (which is a bit dubious), the estimated coefficient for the non-causal variable should tend toward zero and the estimates coefficient on the causal variable should tend toward the true value, as the sample size increases.

2. And if you exclude a variable, you introduce other issues. For instance, you can bias the estimated coefficients. Then, even if you lower the variance-inflation factor and coefficient standard error, the bias may mean a greater amount of error, such as measured by mean squared error.

3. And Harrell’s “fertilizer”$^{\dagger}$ says that parsimony is the enemy of prediction. Sure, prediction is not your explicit goal, but prediction quality factors into coefficient standard errors through the residual variance.

4. Yes, you risk overfitting and lowering AIC and BIC. You also give the model more ability to fit. The AIC/BIC/generalizability are not assured of getting worse upon including additional variables. That’s why we bother to include any variables at all!

I can’t totally get on board with the Wikipedia claim that variable correlation and multicollinearity are not problematic. They are. They lead to large standard errors (variance-inflation factor and related notions). However, eliminating variables is not the self-evidently correct way to handle that, as it might not even help much with shrinking standard errors if the performance is much worse, and there can be an introduction of bias.

Fortunately, large data sets are able to burn through the variance inflation. This makes sense. The reason for variances being inflated is that the mode has trouble telling what to attribute to which of the correlated variables. If high values of a feature correspond with high values of the outcome, that seems like a link, but the other correlated variable also has that kind of relationship with the outcome. To which variable do you attribute the cause?

A large sample size lets you observe the occasional instance where the correlation between the features breaks down. Sure, the two features tend to be high together, but occasionally one will be high while the other is not. If that still results in a high value of the outcome, then you have evidence suggesting a causal link that downplays the impact of the correlated variable.

Think about it like using a microwave. Does the food heat up because you push the button or because microwaves excite the food molecules? If microwaves are basically always emitted when you press the button (high correlation), then you will struggle to determine whether the cause of the heating is the button or the microwave emission. If, however, you manage to emit microwaves some other way and observe the food heating up despite the lack of a button push, then you have evidence that the microwaves do the heating, not your finger. Similarly, if the appliance breaks and does not emit microwaves when I push the button, then the food does not heat up, and we have further evidence supporting microwaves as the cause of the heating instead of pushing the button.

(If you don't like the "dominos" of button push > microwave emission > food warms up, consider that pushing the microwave button also turns the plate inside the microwave. Is the food warmed up by the plate rotation or by the microwaves? When both happen, it is hard to say. However, if the turntable breaks and I notice the food warming up, that is evidence in favor of the microwaves being the cause, especially if I also observe the microwave emittor breaking while the turntable works and the food does not warm up.)

When the variables are correlated, you have to wait a while to observe these deviations from the usual, however, hence the desire for a large sample size that will have such instances.

$^{\dagger}$If Occam’s razor wants to cut down variables, then fertilizer wants to grow them back.

Answer 3

There are a lot of unstated assumptions in this post, so I'll try to be clear about my assumptions in answering. I disagree with the other answers given are argue that multicollinearity is not a problem; it is a fact.

First, I assume your estimand is something like the average causal effect of $A$ on $B$, i.e., $\tau^0 = E[B^{a'}] - E[B^a]$ for two values $a'$ and $a$ of $A$, where $B^a$ is the potential outcome setting $A$ to $a$. If you have some other definition of a causal effect, you need to state it explicitly and in a way that abstracts away from a specific statistical model. I am going to assume this is a prospective observational study in which $A$ is measured before $B$.

There are a variety of estimators of $\tau^0$, but some rely on an assumption called ignorability, or satisfaction of the backdoor criterion, which states that there is a set of measured variables $V$ such that $A \perp B^a | V$ for all $a$. This means that $V$ is a set of variables sufficient to remove confounding. If $V = N$, then you have to adjust for $N$ in its entirety to remove confounding (using population data). This is a fact of nonparametric identification of causal effects and has nothing to do with statistical estimation. In this case, $\tau^0 = \tau \equiv E[E[B|A=a',V]] - E[E[B|A=a',V]]$.

One way to estimate $\tau$ is using an estimator called g-computation. You proposal a model for $B$ as a function of $A$ and $V$, fit that model, then generate predictions $\hat{B^{a'}}$ setting $A$ to $a'$ and predictions $\hat{B^{a}}$ setting $A$ to $a$. You take the mean of $\hat{B^{a'}}$ to be an estimator of $E[E[B|A=a',V]]$ and the mean of $\hat{B^{a}}$ to be an estimator of $E[E[B|A=a,V]]$, and then compute the differences of those two means to get $\hat\tau$. When the model for $B$ is linear with no interaction between $V$ and $A$, then the coefficient on $A$ is equal to $\hat\tau$. I assume this is the estimator you are using.

Let's say $N$ satisfies the backdoor criterion, so you need to adjust for $N$ in its entirety to remove confounding. You find that $N_1$ is highly correlated with $A$. What do you do? First you need to identify the criteria you want to optimize. If you want a consistent estimator of the causal effect (i.e., one that would be closer to correct if the sample were larger), you need to adjust for $V$ (i.e., including $N_1$). If you want a low-error estimate of the causal effect, then you might be willing to sacrifice some bias for precision. There are theories for how the correlation between predictors in a regression model effects how removal of one predictor changes the mean squared error of the other coefficients. Sometimes, a biased but precise estimate is better than an unbiased but imprecise estimate. However, in many causal inference applications, bias is implicitly viewed as more important to reduce (i.e., to maintain objectivity).

I'll address your 4 reasons not to adjust for $N_1$.

1. If $N_1$ causes $A$ and $B$, then you absolutely should adjust for it. Indeed, failing to adjust for it is a failure to satisfy the backdoor criterion. The whole point of causal inference methods is to adjust for confounding variables. A confounding variable is (approximately) a variable that causes both the exposure and outcome. On the other hand, if $A$ causes $N_1$, then $N_1$ is not part of $V$ and adjusting for $N_1$ does not satisfy the backdoor criterion. This is to say that you need to rely on a causal criterion, not a statistical criterion (i.e., collinearity) to decide whether it is necessary to adjust for $N_1$ to satisfy the backdoor criterion.
2. "Unstable" coefficients come with a large standard error; this standard error accurately reflects the uncertainty in estimating the effect of $A$. This uncertainty is a fact, not a decision. That is, a large standard error isn't a reason to change the model to make it smaller, it is a reason to interpret the estimate with appropriate (lack of) certainty.
3. We are not positing theories, we are estimating a causal effect, which is a single parameter. Occam's razor has nothing to do with this. The only quantity Occam's razor acts on is the presence of the causal effect; i.e., it might be more parsimonious to assume there is no causal effect, but that is the reason we are investigating in the first place. To estimate a causal effect consistently, you need to use a consistent estimator. That estimator is not subject to Occam's razor because it is not a theory. It is a recipe that allows you obtain an estimate from data. Parsimony for parsimony's sake (i.e., as an intellectual rather than statistical quality) is irrelevant here. It may be that a more parsimonious model has improved statistical performance (i.e., increased precision), but again this choice depends on the goal of your analysis.
4. If your goal is to estimate a causal effect, why are you concerned with out-of-sample prediction or the fit of the model? Model fit is irrelevant as long as the model permits you to consistently estimate a causal effect. The model isn't designed to explain phenomena; it is designed to play a role in the estimator of the causal effect.

If $N_1$ is necessary to satisfy the backdoor criterion, then omitting it in order to increase precision is saying "I know my estimate will be biased, but the increase in precision due to dropping it is worth the bias incurred." This is a strong claim to make and would need to be application-specific. Knowing the direction of the bias based on the correlation between $N_1$ and $A$ and $B$ could help decide whether your bias is likely to be positive or negative; one or the other may be worse in a given application.

I would say that the reasons you listed seem to come from someone not interested in causal inference but rather interested in building a predictive or explanatory model. Those are very difference goals than estimating a causal effect. To estimate a causal effect, you need to use an estimator that optimizes the criteria you set (e.g., minimizing bias, minimizing mean squared error, interpretability, computational feasibility, etc.), and you need to be assured of the statistical properties of that estimator. An estimator that omits covariates correlated with treatment is likely going to be biased, but it may be more precise. There may be ways to address that bias without reducing precision using more complicated estimators. But you should certainly not take the advice of those building prediction models when your goal is different.