2
$\begingroup$

It's the conventional wisdom that a PCA transformation can cure multicollinearity. Putting this into practice on example data, I find myself confused. In the following case, applying PCA seems to have made the multicollinearity (as measured by VIF) much worse!

library(regclass)
library(car)
library(tidymodels)

data("WINE")

##Dropping rating variable

data<-WINE[,-1]

vif(lm(alcohol~.,data=data))

If we are going by the conservative benchmark that VIF of 2 or greater presents multicollinearity that needs to be removed, we need to do something to remove it. enter image description here

Let's apply PCA.

##3 Varibles with vif over 2, would like to remove that multicollinearity. Let's try pca.

withpca<-recipe(alcohol~.,data=data) %>% step_pca(all_predictors()) %>% prep() %>% juice()



vif(lm(alcohol~.,data=withpca))

#multicollinearity is now much worse!

What has gone wrong? The VIFs for PC1 and PC4 are now astronomical! Can someone explain whether I've misunderstood the conventional wisdom about PCA and multicollinearity or whether that conventional wisdom is just wrong?

enter image description here

$\endgroup$
8
  • $\begingroup$ Your example is too complicated to be useful for studying the effect of PCA in regression. As a result you make programming errors: the new VIFs indeed will all equal $1$ (subject to the effects of floating point imprecision, which can be large in the last few PCs). $\endgroup$
    – whuber
    Feb 1 at 15:19
  • 1
    $\begingroup$ I don't think complexity is the problem, actually. The data set is real, simple, the code is relatively simple...But some applications of PCA scale automatically, some don't. I see now that this application does not automatically scale the data. $\endgroup$ Feb 1 at 16:09
  • $\begingroup$ The code does a lot of things that are irrelevant to the question. This makes it more difficult to diagnose and explain all the things that might be unexpected. $\endgroup$
    – whuber
    Feb 1 at 16:45
  • $\begingroup$ It drops a variable, it finds the vifs of the lm, and then it applies part of the tidymodels package for pre-processing (PCA, scaling). None of that is actually irrelevant. I get that an even simpler approach could help for teaching purposes...But that was not the genesis if this post; this was the code that actually led to the error. $\endgroup$ Feb 1 at 17:31
  • $\begingroup$ That's not a good way to solve a problem, though, nor is it a good way to ask for help with a problem. Please see stackoverflow.com/help/minimal-reproducible-example for some guidance. $\endgroup$
    – whuber
    Feb 1 at 17:39

2 Answers 2

3
$\begingroup$

It seems that I've made a rookie mistake. It's generally recommended to prep the data before PCA by standardizing it first. Some implementations of PCA automatically do so. This one does not, apparently.

withpca<-recipe(alcohol~.,data=data) %>%step_center(all_predictors())  %>% step_scale(all_predictors()) %>% step_pca(all_predictors()) %>% prep() %>% juice()


vif(lm(alcohol~.,data=withpca))

Now the VIFs look perfect.enter image description here

$\endgroup$
6
  • 3
    $\begingroup$ Centering (and in many cases, scaling) should have nothing to do with multicollinearity. $\endgroup$ Feb 1 at 6:19
  • $\begingroup$ Ok, did I say that it did? Centering and scaling impact PCA. That part is clear. $\endgroup$ Feb 1 at 13:28
  • $\begingroup$ You said that centering and scaling solved your problem. This is the second sentence of your answer. $\endgroup$ Feb 1 at 13:43
  • $\begingroup$ I think my meaning is pretty clear--go back and read it. I said that centering and scaling clearly have an impact here on PCA. In fact, in many places (as I'm sure you know), it is recommended that you center/scale your data before performing PCA in (this particular implementation)...The whole point is that PCA does in fact address multicollinearity. $\endgroup$ Feb 1 at 13:59
  • $\begingroup$ I do not think we disagree on what you have written. My comment addresses the validity of that. $\endgroup$ Feb 1 at 14:09
1
$\begingroup$

Principal Components Analysis is sensitive to the scaling of the variables (variables measured on different scales and with different variances). Scaling is recommended when computing PCs by singular value decomposition (see stats::prcomp).
Now, compute the principal components for your data with and without scaling and then inspect the loadings for PC1 ('rotation' for stats::prcomp). Without scaling, the variable with the largest variance has the largest loading. With scaling, the loadings will be quite different.
Next, look at the bivariate correlations between the variables and the principal components (best done with scatterplots). That will show you which PCs are strongly correlated with which variables. However, for some data I am looking at, the PCs aren't strongly correlated with one another (they shouldn't) but then bivariate correlations are not necessarily reliable indicators of variance inflation. Also keep in mind that VIFs change as you drop variables.
Finally, you need to consider if the PCs have a meaningful interpretation for communicating the the results of your model. It might be better to use the original variables and drop those with large VIFs in a step wise manner.
I'm (obviously) not a statistician but it seems that you did make a rookie mistake in not scaling the variables for PCA.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.