11
$\begingroup$

I know that someone will probably say that this question is repeated and I will get a negative vote, but I'm very convinced that it's not, or at least it wasn't properly answered. See, we have lots of questions like this, sure. For instance, this one, also this one, this, and also this post, oh and also this post.

By reading all the answers, what I've found is that most people believe that correlation and multicollinearity are the same, or very related, but some very prestigious people (like Peter Flom) says that they are not actually directly related. Also, most answers are not very profound. At least not for someone not so intelligent like me. And there is nothing in the books I have as well (like Angrist and Wooldridge).

So I will take my chances here and ask again: what is the difference between multicollinearity and correlation? How do I check mathematically multicollinearity? What kind of math is behind this?

I'm trying to 'see' the difference mathematically. Can someone help me?

$\endgroup$
2
  • 5
    $\begingroup$ Correlation is about two variables whereas multicolinearity can be about linear function of arbitrarily many variables. Apart from that I look forward to read some answer. +1 $\endgroup$
    – Bernhard
    Sep 18 at 20:19
  • 1
    $\begingroup$ I believe you misrepresent Peter Flom's answer. Although the relationship between correlation and multicollinearity may be somewhat subtle, it is direct. Your questions are answered in the links. For instance, one of them states you can look at the VIF to check for multicollinearity. The math is called "linear algebra." Because there is no essential difference--apart from what was just pointed out in Bernhard's comment--there's nothing to see mathematically. $\endgroup$
    – whuber
    Sep 18 at 21:17
9
$\begingroup$

According to the Wikipedia encyclopedia

In statistics, multicollinearity (also collinearity) is a phenomenon in which one predictor variable in a multiple regression model can be linearly predicted from the others with a substantial degree of accuracy.

So multicollinearity is a special case of correlation. It is more specific in two ways

  • It relates specifically to the predictor variables in a regression model.
  • It relates to correlation with a linear relationship of multiple variables combined.

Correlation doesn't need to be that linear relationship of multiple variables. And correlation describes a much wider phenomena than just this case with predictor variables.

There is a distinction between perfect multicollinearity (a perfect linear relation between variables/predictors) or just multicollinearity (not an exact linear relationship but strong correlation of at least one of the predictors with a linear combination of the other predictors).


Is checking for multicollinearity the same as checking for correlation?

No, not exactly.

what is the difference between multicollinearity and correlation? How do I check mathematically multicollinearity? What kind of math is behind this?

Multicollinearity may occur even when there is little correlation present between individual pairs of predictors. The issue of multicollinearity can occur when there is correlation with one predictor and a linear sum of the other predictors.

Imagine for instance when there are six predictors and a seventh is added as the sum $X_7 = X_1 + X_2 + X_ 3 + X_4 + X_5 + X_6$. The correlation between $X_7$ and the other predictors will only be relatively small. But there will be perfect multicollinearity.

In that example of perfect linear relationship, you can check perfect multicollinearity by computing the rank of the design matrix, and this should equal the number of columns in order for perfect multicollinearity to be absent.

But when $X_7$ has just a tiny bit of difference with the sum $X_1 + X_2 + X_3 + X_4 + X_5 + X_6$ then there won't be perfect multicollinearity. Yet, the predictor $X_7$ is still a lot dependent on the other six (and causing troubles).

In this case you can not easily verify just by the correlations, because these are not very large. One common method, in this case, is to compute the variance inflation factor (VIF). This VIF expresses how much of the variation/variance/error in the estimate of a coefficient (computed as $s^2 (X^TX)^{-1}$, where $X$ is the design matrix) is due to the interactions with the other variables.

Computational example

Below we created the seven variables as explained above. The seventh is the mean of the previous six with a bit of noise added.

The correlation table (of the design matrix $X$) looks like:

 1.00 -0.30 -0.25 -0.32  0.11 -0.24 0.01
-0.30  1.00  0.29  0.29 -0.12  0.10 0.64
-0.25  0.29  1.00  0.11 -0.54  0.34 0.41
-0.32  0.29  0.11  1.00  0.03 -0.17 0.46
 0.11 -0.12 -0.54  0.03  1.00 -0.44 0.10
-0.24  0.10  0.34 -0.17 -0.44  1.00 0.28
 0.01  0.64  0.41  0.46  0.10  0.28 1.00

Not very remarkable.

But the inverse of the covariance table $(X^TX)^{-1}$. Has the last entry on the diagonal a large value 6.42. This relates to the error (variance) of the 7th coefficient which is almost 36 times larger/inflated.

 0.15  0.17  0.15  0.15  0.17  0.13 -0.93
 0.17  0.24  0.18  0.18  0.23  0.16 -1.15
 0.15  0.18  0.22  0.16  0.22  0.14 -1.07
 0.15  0.18  0.16  0.21  0.20  0.15 -1.06
 0.17  0.23  0.22  0.20  0.29  0.19 -1.29
 0.13  0.16  0.14  0.15  0.19  0.16 -0.93
-0.93 -1.15 -1.07 -1.06 -1.29 -0.93  6.42

Below is the table of the design matrix

x_1 x_2 x_3 x_4 x_5 x_6 6 x_7
4 7 6 7 4 5 5.67
4 3 5 4 5 4 4.17
5 5 6 4 4 4 4.67
6 3 5 4 4 8 4.83
3 4 5 5 6 5 4.83
6 4 6 4 3 3 4.33
7 1 2 4 5 3 3.67
5 4 4 6 3 4 4.17
5 6 5 2 3 7 4.50
2 4 5 6 3 5 4.17
3 4 4 4 3 7 4.17
3 5 6 6 2 5 4.33
5 4 4 4 5 4 4.17
4 3 3 4 6 4 4.00
6 6 2 4 6 3 4.67
4 5 3 6 6 1 4.17
5 6 4 6 4 5 5.00
8 3 5 4 4 3 4.50
4 5 5 6 6 4 5.17
6 4 4 7 5 5 5.17

The example is created with this R-code

set.seed(1)

# generate 6 random variables
X_1_6  = matrix(rbinom(120,9,0.5), ncol = 6)

# generate a 7-th variable with a bit noise
x7 = (rowSums(X_1_6) + rbinom(20, 2, 0.5) - 1)/6

# make the design matrix from all 7 variables
X = cbind(X_1_6,x7)

### the correlations
round(cor(X), 2)

### the inverse of the covariance table
round(solve(t(X) %*% X),2)

Geometric view

I'm trying to 'see' the difference mathematically.

From a geometric point of view you can see regression as projection of the $n$ observations $y$ onto the surface spanned by the $m$ predictor vectors.

These observations can be seen as a point in $n$-dimensional space. The $m$ predictor vectors form a sub-space (m-dimensional, in 2 dimensions you can see it as a surface) within this $n$-dimensional space. The fitted solution of the regression is the point within this sub-space that is closest to the observation.

The image below from this question might help to see this.

illustration for a small sample size

We could look perpendicular to the surface spanned by the vectors $x_1$ and $x_2$ above. The coefficients $\beta_1$ and $\beta_2$ of the solution can be seen as coordinates on this surface telling how much of vector $x_1$ and $x_2$ you need to add to get to the solution/prediction $\hat{y}$. (see the question Intuition behind $(X^TX)^{-1}$ in closed form of w in Linear Regression for an explanation of the difference between the $\alpha$ and $\beta$ coordinates in this image)

illustration for a small sample size

The axes of this coordinate space are however not perpendicular to each other. The following image illustrates that this creates large changes in coordinates for small changes of a point in space.

narrow coordinates

The image contains a sample of 100 randomly distributed observations/experiments as the projections/solutions of the fitting/regression. The distribution of this is a circular cloud in the case of Gaussian distributed errors.

The axes $x_1$ and $x_2$ are not perpendicular to each other but are diagonal instead. You can see that this sort of places the lines of the coordinates closer to each other (and these coordinates correspond to the coefficients that are the output of the regression). This means that the variation in this coordinate/coefficient will be larger.

The case of this graphical example is in two dimensions, but you could imagine it as extended to multiple dimensions. Multicollinearity means that at least one of the axes (corresponding to the predictor vector) has a small angle with the other axes combined. That means it is not necessarily one single axis being close to another single axis (in the 2D case it is), but one axis being close to the subspace created by the other $m-1$ other axes.

In 3 dimensions you can view it as below. Say the axis $x_3$ is sheared and at an angle with the others. You can have this $x_3$ at a very small angle with the bottom plane of the cube (or even inside of it) without the independent angles with $x_1$ and $x_2$ being very small.

3D example


See for a related question (with many more links to other related questions) why does the same variable have a different slope when incorporated into a linear model with multiple x variables with

$\endgroup$
4
  • 1
    $\begingroup$ "Galton defined what co-relation as a phenomenon that occurs when "the variation of the one [variable] is accompanied on the average by more or less variation of the other, and in the same direction."" See wiki. $\endgroup$
    – Galen
    Sep 19 at 4:14
  • $\begingroup$ This post is confusing. Even in "linear relationships" pairs of variables often are non-linearly related; and to state that "correlation doesn't need to be a linear relationship" is puzzling, because correlation isn't a relationship at all. The rest is even more confusing because it contradicts itself. "Multicollinearity may occur even when there is little correlation present" means, geometrically, the angles between the axes are all large, but later you assert "Multicollinearity means that at least one of the axes ... has a small angle with the other axes." How can both be correct? $\endgroup$
    – whuber
    Sep 19 at 19:07
  • 2
    $\begingroup$ @whuber I have rephrased it a bit. The point that I tried to convey is that the correlation between individual pairs of predictors is not what makes the multicollinearity, but it is the correlation between linear combinations of predictors. $\endgroup$ Sep 19 at 19:43
  • $\begingroup$ I get it, but unless you constrain the possible linear combinations, the latter is meaningless. $\endgroup$
    – whuber
    Sep 20 at 12:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.