Why not use PCA in every linear regression setting to avoid multicollinearity?

Question

I realise the stupidity of this question, but hear me out.

Imagine a linear regression (e.g. OLS) setting where we perform PCA on all of our independent variables and use all of the resulting principal components (so, number of principal components = number of independent variables) as independent variables instead. From a glance it looks like this will rectify any multicollinearity problem no matter how severe (since all principal components are orthogonal). Moreover, we will not have any problems interpreting the resulting coefficients since they can be projected back into the original coefficient space with no identity loss (since number of principal components we used = number of independent variables) by simply taking a product with principal components. So, a win-win.

Obviously something is wrong with this approach. I just can't get my head around a framework to analyze such situation properly. Any help is appreciated.

EDIT:

To put the question into a perspective, imagine you are tasked with building an OLS model. There are two independent variables which are highly collinear. Usually you would either leave one of them out or perform PCA on them and use the first principal component as a predictor. However, the person who tasked you with building the model is interested in estimating coefficients on both of those variables, so the aforementioned options are not available. Moreover, this person wants the model to meet certain formal criteria, one of which is VIFs below a certain threshold, which is not possible if you include the two highly collinear independent variables as is. Therefore, the only option left is to use PCA on these two variables and use the resulting two principal components as predictors, which allows us to obtain coefficients on both of them in the original coefficient space (by multiplying the coefficients for principal components and principal components themselves) and pass VIF threshold. The person who tasked you with building the model is perfectly fine with this approach, but I can't help but feel that something is off. Are we simply masking the multicollinearity problem here to fool the VIF metric?

Answer 1

This is probably a comment rather than an answer and I'm not sure I'm getting it right. Let's try to compare the output of a linear model on two nearly colinear variables before and after PCA:

set.seed(1234)
x1 <- 1:10
x2 <- x1 + rnorm(n= length(x1), sd= 0.0001) # x2 is nearly colinear to x1

y <- rowMeans(cbind(x1, x2)) + rnorm(n= length(x1)) # A response variable

Linear regression on raw data:

summary(lm(y ~ x1 + x2))
...    
Coefficients:
              Estimate Std. Error t value Pr(>|t|)
(Intercept)    -0.9351     0.7503  -1.246    0.253
x1           1428.6673  3681.8475   0.388    0.710
x2          -1427.5288  3681.8604  -0.388    0.710

Residual standard error: 1.094 on 7 degrees of freedom
Multiple R-squared:  0.9281,    Adjusted R-squared:  0.9076 
F-statistic: 45.18 on 2 and 7 DF,  p-value: 9.963e-05

Now on the principal components:

pca <- prcomp(cbind(x1, x2))

pca_lm <- lm(y ~ pca$x[,1] + pca$x[,2])
summary(pca_lm)
...
Coefficients:
             Estimate Std. Error t value Pr(>|t|)    
(Intercept) 5.382e+00  3.458e-01  15.562 1.09e-06 ***
pca$x[, 1]  8.086e-01  8.514e-02   9.498 3.00e-05 ***
pca$x[, 2]  2.020e+03  5.207e+03   0.388     0.71    
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 1.094 on 7 degrees of freedom
Multiple R-squared:  0.9281,    Adjusted R-squared:  0.9076 
F-statistic: 45.18 on 2 and 7 DF,  p-value: 9.963e-05

Looking at adjusted R-squared, the quality of the two models is the same - as expected.

Project the coefficients from model with principal components to the original scale (am I doing this right?)

(pca_lm$coefficients[1] + pca_lm$coefficients[2:3]) %*% pca$rotation
          PC1       PC2
[1,] 1436.278 -1427.529

These are similar to the coefficients from the model on raw variables. So, in summary, there is no advantage in passing by principal components.

Answer 2

In this answer, I will be using the example given in dariober's answer. There are two variables x1 and x2 that differ from each other only by a small noise (sd= 0.0001). The response is also correlated to the two, although by a somewhat larger noise (no sd specified, defaults to 1). Now, we would expect that the two variables to give somewhat equal contributions to the linear model. Ideally, it will be around 0.5 each.

What we get instead are very large coefficients that are similar in magnitude but with opposing signs. This is because they are trying to fit the noise in the response, and the difference between the variables, which is noise, may have some correlations with the noise in the response.

By applying PCA, the highly correlated component of the two variables are put into PC1, while the uncorrelated component are put into PC2. By applying regression on these two, you now see the true contribution of the highly correlated component, while the uncorrelated component has a component with very large magnitude and very large standard error.

Now, if you use all the PCs then you have not solved the problem with multicollinearity. The next step is to remove the problematic PC2 and recover the coefficients for the original variables by rotating the coefficients back (though there's only one of them now). After removal of PC2, the final coefficients for x1 and x2 were 0.5718034 and 0.5718014 respectively, which is in agreement to the original intuition we originally had in the first paragraph.