The definition of multicollinearity is:

Given a set of $N \times 1$ predictors $X = (x_1, x_2, \cdots, x_m)$, if $$x_j = \sum_{i \neq j}a_ix_i$$ then we say there is multicollinearity among the predictors.

I tend to interpret $x_j = \sum_{i \neq j}a_ix_i$ in such a way:

The variation along the $jth$ direction is explained by those directions whose $a_i$ is non-zero. If the sum can be interpreted in such a way, then by analyzing the eigenvalues of $X^TX$, we can determine whether there is multicollinearity.

Am I right?


I am not sure you are right. You very well may be. Just a couple of things to keep in mind.

PCA is explicitly structured as to resolve multicollinearity among multiple variables. Given that, it is a very useful tool to circumvent multicollinearity. On the other hand, it is not the most efficient test to diagnose multicollinearity.

To test for multicollinearity, it is a lot more efficient to use the traditional Tolerance and Variance Inflation Factor (VIF) indicators that are readily generated by most software.

Just to cover the basics, you can calculate a lot of that stuff long hand easily. Let's say the correlation between two variables is very high at 0.95. In turn R Square is 0.95^2 = 0.90. The Tolerance is 1 - R^2. In this case, it is 1 - 0.90 = 0.10. And, the VIF is 1/Tolerance or 1/0.10 = 10. An interesting measure is the square root of the variance, in this case ~ 3. What the latter is saying is that if you use a linear regression with the original Y and the X variable that has a SQRT(VIF) of 3, the standard error of X will be 3 times larger within your multiple regression model vs the linear regression you just ran with only X. But, that is maybe not such a problem if X is already statistically significant.

As reviewed the statistical framework of multicollinearity testing as depicted above is far more transparent than PCA. Given that, I would never go through the unnecessary trouble of using PCA to test for multicollinearity. This does not mean that interpreting multicollinearity between variables with PCA as you outline is wrong.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.