11
$\begingroup$

I realise the stupidity of this question, but hear me out.

Imagine a linear regression (e.g. OLS) setting where we perform PCA on all of our independent variables and use all of the resulting principal components (so, number of principal components = number of independent variables) as independent variables instead. From a glance it looks like this will rectify any multicollinearity problem no matter how severe (since all principal components are orthogonal). Moreover, we will not have any problems interpreting the resulting coefficients since they can be projected back into the original coefficient space with no identity loss (since number of principal components we used = number of independent variables) by simply taking a product with principal components. So, a win-win.

Obviously something is wrong with this approach. I just can't get my head around a framework to analyze such situation properly. Any help is appreciated.

EDIT:

To put the question into a perspective, imagine you are tasked with building an OLS model. There are two independent variables which are highly collinear. Usually you would either leave one of them out or perform PCA on them and use the first principal component as a predictor. However, the person who tasked you with building the model is interested in estimating coefficients on both of those variables, so the aforementioned options are not available. Moreover, this person wants the model to meet certain formal criteria, one of which is VIFs below a certain threshold, which is not possible if you include the two highly collinear independent variables as is. Therefore, the only option left is to use PCA on these two variables and use the resulting two principal components as predictors, which allows us to obtain coefficients on both of them in the original coefficient space (by multiplying the coefficients for principal components and principal components themselves) and pass VIF threshold. The person who tasked you with building the model is perfectly fine with this approach, but I can't help but feel that something is off. Are we simply masking the multicollinearity problem here to fool the VIF metric?

$\endgroup$
21
  • 1
    $\begingroup$ The simple answer is because data is rarely nice. The dominant sources of variation are not always related to the independent variable. $\endgroup$
    – ReneBt
    Commented Nov 9, 2020 at 10:31
  • $\begingroup$ Additionally, it is just inconvenient. It is a lot of extra effort and fiddling around for no real gain in performance. $\endgroup$ Commented Nov 9, 2020 at 10:35
  • 4
    $\begingroup$ Much depends also on the goals of the exercise. In many fields, we should want to think about a regression in terms of underlying processes (use whatever term suits: attitudes and behaviour might fit, e.g). We also want to compare our study with previous studies (and indeed trust that our study will be interesting and useful for later workers). A regression with PCs as predictors is much harder to think about than one with predictors that are named and which can be related to whatever natural scientific, social scientific, clinical, engineering or business knowledge can be brought to bear. $\endgroup$
    – Nick Cox
    Commented Nov 9, 2020 at 10:57
  • 2
    $\begingroup$ Scaring readers about multicollinearity is a bad tradition passed down from old textbooks written in the times when solving the equations was a big deal. There needs to be advice about predictor choice, surely, but bad predictor choice is evident in the results. $\endgroup$
    – Nick Cox
    Commented Nov 9, 2020 at 12:16
  • 1
    $\begingroup$ @Nick Cox I'm not trying to argue about the significance of multicollinearity problem or about the best practices of analysis of real world data. I'm simply trying to understand if there's anything wrong with the logic I've laid out from theoretical standpoint. Since «going from A to B via C» like you described seems to eliminate multicollinearity as a byproduct, I'm trying to understand whether this is the case, and if not, why. $\endgroup$
    – Armadillo
    Commented Nov 9, 2020 at 12:22

2 Answers 2

2
$\begingroup$

This is probably a comment rather than an answer and I'm not sure I'm getting it right. Let's try to compare the output of a linear model on two nearly colinear variables before and after PCA:

set.seed(1234)
x1 <- 1:10
x2 <- x1 + rnorm(n= length(x1), sd= 0.0001) # x2 is nearly colinear to x1

y <- rowMeans(cbind(x1, x2)) + rnorm(n= length(x1)) # A response variable

Linear regression on raw data:

summary(lm(y ~ x1 + x2))
...    
Coefficients:
              Estimate Std. Error t value Pr(>|t|)
(Intercept)    -0.9351     0.7503  -1.246    0.253
x1           1428.6673  3681.8475   0.388    0.710
x2          -1427.5288  3681.8604  -0.388    0.710

Residual standard error: 1.094 on 7 degrees of freedom
Multiple R-squared:  0.9281,    Adjusted R-squared:  0.9076 
F-statistic: 45.18 on 2 and 7 DF,  p-value: 9.963e-05

Now on the principal components:

pca <- prcomp(cbind(x1, x2))

pca_lm <- lm(y ~ pca$x[,1] + pca$x[,2])
summary(pca_lm)
...
Coefficients:
             Estimate Std. Error t value Pr(>|t|)    
(Intercept) 5.382e+00  3.458e-01  15.562 1.09e-06 ***
pca$x[, 1]  8.086e-01  8.514e-02   9.498 3.00e-05 ***
pca$x[, 2]  2.020e+03  5.207e+03   0.388     0.71    
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 1.094 on 7 degrees of freedom
Multiple R-squared:  0.9281,    Adjusted R-squared:  0.9076 
F-statistic: 45.18 on 2 and 7 DF,  p-value: 9.963e-05

Looking at adjusted R-squared, the quality of the two models is the same - as expected.

Project the coefficients from model with principal components to the original scale (am I doing this right?)

(pca_lm$coefficients[1] + pca_lm$coefficients[2:3]) %*% pca$rotation
          PC1       PC2
[1,] 1436.278 -1427.529

These are similar to the coefficients from the model on raw variables. So, in summary, there is no advantage in passing by principal components.

$\endgroup$
2
  • $\begingroup$ The «advantage» is that the model built on principal components supposedly does not suffer from multicollinearity since principal components are orthogonal. I want to know whether this is the case and if not, why. $\endgroup$
    – Armadillo
    Commented Nov 9, 2020 at 12:08
  • $\begingroup$ +1 I'm curious about what the PCA -> original scale transformation implies for the parameter uncertainties. $\endgroup$
    – mkt
    Commented Aug 24, 2023 at 18:06
0
$\begingroup$

In this answer, I will be using the example given in dariober's answer. There are two variables x1 and x2 that differ from each other only by a small noise (sd= 0.0001). The response is also correlated to the two, although by a somewhat larger noise (no sd specified, defaults to 1). Now, we would expect that the two variables to give somewhat equal contributions to the linear model. Ideally, it will be around 0.5 each.

What we get instead are very large coefficients that are similar in magnitude but with opposing signs. This is because they are trying to fit the noise in the response, and the difference between the variables, which is noise, may have some correlations with the noise in the response.

By applying PCA, the highly correlated component of the two variables are put into PC1, while the uncorrelated component are put into PC2. By applying regression on these two, you now see the true contribution of the highly correlated component, while the uncorrelated component has a component with very large magnitude and very large standard error.

Now, if you use all the PCs then you have not solved the problem with multicollinearity. The next step is to remove the problematic PC2 and recover the coefficients for the original variables by rotating the coefficients back (though there's only one of them now). After removal of PC2, the final coefficients for x1 and x2 were 0.5718034 and 0.5718014 respectively, which is in agreement to the original intuition we originally had in the first paragraph.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.