Can PCA be used for detecting multicollinearity?

Question

The definition of multicollinearity is:

Given a set of $N \times 1$ predictors $X = (x_1, x_2, \cdots, x_m)$, if $$x_j = \sum_{i \neq j}a_ix_i$$ then we say there is multicollinearity among the predictors.

I tend to interpret $x_j = \sum_{i \neq j}a_ix_i$ in such a way:

The variation along the $jth$ direction is explained by those directions whose $a_i$ is non-zero. If the sum can be interpreted in such a way, then by analyzing the eigenvalues of $X^TX$, we can determine whether there is multicollinearity.

Am I right?

Answer 1

I am not sure you are right. You very well may be. Just a couple of things to keep in mind.

PCA is explicitly structured as to resolve multicollinearity among multiple variables. Given that, it is a very useful tool to circumvent multicollinearity. On the other hand, it is not the most efficient test to diagnose multicollinearity.

To test for multicollinearity, it is a lot more efficient to use the traditional Tolerance and Variance Inflation Factor (VIF) indicators that are readily generated by most software.

Just to cover the basics, you can calculate a lot of that stuff long hand easily. Let's say the correlation between two variables is very high at 0.95. In turn R Square is 0.95^2 = 0.90. The Tolerance is 1 - R^2. In this case, it is 1 - 0.90 = 0.10. And, the VIF is 1/Tolerance or 1/0.10 = 10. An interesting measure is the square root of the variance, in this case ~ 3. What the latter is saying is that if you use a linear regression with the original Y and the X variable that has a SQRT(VIF) of 3, the standard error of X will be 3 times larger within your multiple regression model vs the linear regression you just ran with only X. But, that is maybe not such a problem if X is already statistically significant.

As reviewed the statistical framework of multicollinearity testing as depicted above is far more transparent than PCA. Given that, I would never go through the unnecessary trouble of using PCA to test for multicollinearity. This does not mean that interpreting multicollinearity between variables with PCA as you outline is wrong.