Return to Question

Suppose we are inferring whether $A$ causes $B$, while holding $N = [N_0, N_1, \ldots, N_n]$ constant and we find $N_i$ correlates well with $A$. There are four reasons to exclude $N_i$:

1. We may have discovered a causal confound $N_i \rightarrow A$ (read: $N_i$ causes $A$) or $A \rightarrow N_i$. If there is a plausible causal relationship (either direction), then $N_i$ should not be controlled for, because it is a confounder ($N_i \rightarrow A$ and $N_i \rightarrow B$), a mediator ($A \rightarrow N_i \rightarrow B$).

2. Besides frustrating causal inference, the resulting coefficients will be unstable; doing the experiment again might have very different coefficients for $A$ and $N_i$, meaning the result and effect-size may not be reproducible.

3. Occam's Razor suggests a more parsimonious theory (that with $N_i$ removed) should be pursued first. If $A$ and $N_i$ are collinear, then a theory with just $A$ will be more parsimonious and almost as good as a theory with $A$ and $N_i$.

4. Having that additional coefficient (free parameter) also increases the chance of overfitting, since there are more degrees of freedom, which worsens AIC, BIC, and out-of-sample generalization.

So, why does Wikipedia claim multicollinearity among predictors is not a problem (multiple times in that article) and not a reason to exclude variables, stating

High collinearity indicates that it is exceptionally important to include all collinear variables, as excluding any will cause worse coefficient estimates, strong confounding, and downward-biased estimates of standard errors.

I agree that blindly removing them post hoc is unjustified, but it should warn the researcher to re-check their a priori reasoning for choosing those variables in the first place, assessing possibility of a mediating and confounding effect and whether the additional parameter is "worth it" in its predictive value.

Suppose we are inferring whether $A$ causes $B$, while holding $N = [N_0, N_1, \ldots, N_n]$ constant and we find $N_i$ correlates well but not perfectly with $A$. There are four reasons to exclude $N_i$:

1. We may have discovered a causal confound $N_i \rightarrow A$ (read: $N_i$ causes $A$) or $A \rightarrow N_i$. If there is a plausible causal relationship (either direction), then $N_i$ should not be controlled for, because it is a confounder ($N_i \rightarrow A$ and $N_i \rightarrow B$), a mediator ($A \rightarrow N_i \rightarrow B$).

2. The resulting coefficients will be unstable and potentially irreproducible; doing the experiment again might select a different pair of coefficients for $A$ and $N_i$ (I'm not assuming the correlation is perfect; just good enough to, with a sprinkling of statistical noise, select different coefficients).

3. Occam's Razor suggests a more parsimonious theory (that with $N_i$ removed) should be pursued first. If $A$ and $N_i$ are collinear, then a theory with just $A$ will be more parsimonious and almost as good as a theory with $A$ and $N_i$.

4. Having that additional coefficient (free parameter) also increases the chance of overfitting, since there are more degrees of freedom, which worsens AIC, BIC, and out-of-sample generalization.

So, why does Wikipedia claim multicollinearity among predictors is not a problem (multiple times in that article) and not a reason to exclude variables, stating

High collinearity indicates that it is exceptionally important to include all collinear variables, as excluding any will cause worse coefficient estimates, strong confounding, and downward-biased estimates of standard errors.

I agree that blindly removing them post hoc is unjustified, but it should warn the researcher to re-check their a priori reasoning for choosing those variables in the first place, assessing possibility of a mediating and confounding effect and whether the additional parameter is "worth it" in its predictive value.

Suppose we are inferring whether $A$ causes $B$, while holding $N = [N_0, N_1, \ldots, N_n]$ constant and we find $N_i$ correlates well with $A$. There are four reasons to exclude $N_i$:

1. We may have discovered a causal confound $N_i \rightarrow A$ (read: $N_i$ causes $A$) or $A \rightarrow N_i$. If there is a plausible causal relationship (either direction), then $N_i$ should not be controlled for, because it is a confounder ($N_i \rightarrow A$ and $N_i \rightarrow B$), a mediator ($A \rightarrow N_i \rightarrow B$).

2. Besides frustrating causal inference, the resulting coefficients will be unstable; doing the experiment again might have very different coefficients for $A$ and $N_i$, meaning the result and effect-size may not be reproducible.

3. Occam's Razor suggests a more parsimonious theory (that with $N_i$ removed) should be pursued first. If $A$ and $N_i$ are collinear, then a theory with just $A$ will be more parsimonious and almost as good as a theory with $A$ and $N_i$.

4. Having that additional coefficient (free parameter) also increases the chance of overfitting, since there are more degrees of freedom, which worsens AIC, BIC, and out-of-sample generalization.

So, why does Wikipedia claim multicollinearity among predictors is not a problem (multiple times in that article) and not a reason to exclude variables, stating

High collinearity indicates that it is exceptionally important to include all collinear variables, as excluding any will cause worse coefficient estimates, strong confounding, and downward-biased estimates of standard errors.

I agree that blindly removing them post hoc is unjustified, but it should warn the researcher to re-check their a priori reasoning for choosing those variables in the first place, assessing possibility of a mediating and confounding effect and whether the additional parameter is "worth it" in its predictive value.

Suppose we are inferring whether $A$ causes $B$, while holding $N = [N_0, N_1, \ldots, N_n]$ constant and we find $N_i$ correlates well with $A$. There are four reasons to exclude $N_i$:

1. We may have discovered a causal confound $N_i \rightarrow A$ (read: $N_i$ causes $A$) or $A \rightarrow N_i$. If there is a plausible causal relationship (either direction), then $N_i$ should not be controlled for, because it is a confounder ($N_i \rightarrow A$ and $N_i \rightarrow B$), a mediator ($A \rightarrow N_i \rightarrow B$).

2. Besides frustrating causal inference, the resulting coefficients will be unstable; doing the experiment again might have very different coefficients for $A$ and $N_i$, meaning the result and effect-size may not be reproducible.

3. Occam's Razor suggests a more parsimonious theory (that with $N_i$ removed) should be pursued first. If $A$ and $N_i$ are collinear, then a theory with just $A$ will be more parsimonious and almost as good as a theory with $A$ and $N_i$.

4. Having that additional coefficient (free parameter) also increases the chance of overfitting, since there are more degrees of freedom, which worsens AIC, BIC, and out-of-sample generalization.

So, why does Wikipedia claim multicollinearity among predictors is not a problem (multiple times in that article) and not a reason to exclude variables, stating

High collinearity indicates that it is exceptionally important to include all collinear variables, as excluding any will cause worse coefficient estimates, strong confounding, and downward-biased estimates of standard errors.

I agree that blindly removing them post hoc is unjustified, but it should warn the researcher to re-check their a priori reasoning for choosing those variables in the first place, assessing possibility of a mediating and confounding effect and whether the additional parameter is "worth it" in its predictive value.

Suppose we are inferring whether $A$ causes $B$, while holding $N = [N_0, N_1, \ldots, N_n]$ constant. If we find any $N_i \in N$ correlates well with $A$, then we may have discovered a causal confound $N_i \rightarrow A$ (read: $N_i$ causes $A$) or $A \rightarrow N_i$. In either case, the researcher should consider the a priori relationship between $A$ and $N_i$. If there is a plausible causal relationship (either direction), then $N_i$ should not be controlled for, because it is a confounder ($N_i \rightarrow A$ and $N_i \rightarrow B$), a mediator ($A \rightarrow N_i \rightarrow B$), or redundant ($A \rightarrow B$ and $A \rightarrow N_i$). In any case, regressing $B$ on $[A, N_0, N_1 \ldots, N_i]$ may return a zero coefficient for $A$ and a non-zero coefficient for $N_i$ or vice versa, I think.

Besides frustrating causal inference, the resulting coefficients will be unstable; doing the experiment again might have very different coefficients for $A$ and $N_i$, meaning the result and effect-size may not be reproducible.

So, why does Wikipedia claim multicollinearity among predictors is not a problem (multiple times in that article) and not a reason to exclude variables, stating

High collinearity indicates that it is exceptionally important to include all collinear variables, as excluding any will cause worse coefficient estimates, strong confounding, and downward-biased estimates of standard errors.

I agree that blindly removing them post hoc is unjustified, but it should warn the researcher to re-check their a priori reasoning for choosing those variables in the first place, assessing possibility of a mediating and confounding effect.

Suppose we are inferring whether $A$ causes $B$, while holding $N = [N_0, N_1, \ldots, N_n]$ constant and we find $N_i$ correlates well with $A$. There are four reasons to exclude $N_i$:

1. We may have discovered a causal confound $N_i \rightarrow A$ (read: $N_i$ causes $A$) or $A \rightarrow N_i$. If there is a plausible causal relationship (either direction), then $N_i$ should not be controlled for, because it is a confounder ($N_i \rightarrow A$ and $N_i \rightarrow B$), a mediator ($A \rightarrow N_i \rightarrow B$).

2. Besides frustrating causal inference, the resulting coefficients will be unstable; doing the experiment again might have very different coefficients for $A$ and $N_i$, meaning the result and effect-size may not be reproducible.

3. Occam's Razor suggests a more parsimonious theory (that with $N_i$ removed) should be pursued first. If $A$ and $N_i$ are collinear, then a theory with just $A$ will be more parsimonious and almost as good as a theory with $A$ and $N_i$.

4. Having that additional coefficient (free parameter) also increases the chance of overfitting, since there are more degrees of freedom, which worsens AIC, BIC, and out-of-sample generalization.

So, why does Wikipedia claim multicollinearity among predictors is not a problem (multiple times in that article) and not a reason to exclude variables, stating

High collinearity indicates that it is exceptionally important to include all collinear variables, as excluding any will cause worse coefficient estimates, strong confounding, and downward-biased estimates of standard errors.

I agree that blindly removing them post hoc is unjustified, but it should warn the researcher to re-check their a priori reasoning for choosing those variables in the first place, assessing possibility of a mediating and confounding effect and whether the additional parameter is "worth it" in its predictive value.