# Does standardising independent variables reduce collinearity?

I've come across a very good text on Bayes/MCMC. IT suggests that a standardisation of your independent variables will make an MCMC (Metropolis) algorithm more efficient, but also that it may reduce (multi)collinearity. Can that be true? Is this something I should be doing as standard.(Sorry).

Kruschke 2011, Doing Bayesian Data Analysis. (AP)

Thanks all for your time and helpful comments

R

edit: for example

     > data(longley)
> cor.test(longley$Unemployed, longley$Armed.Forces)

Pearson's product-moment correlation

data:  longley$Unemployed and longley$Armed.Forces
t = -0.6745, df = 14, p-value = 0.5109
alternative hypothesis: true correlation is not equal to 0
95 percent confidence interval:
-0.6187113  0.3489766
sample estimates:
cor
-0.1774206

> standardise <- function(x) {(x-mean(x))/sd(x)}
> cor.test(standardise(longley$Unemployed), standardise(longley$Armed.Forces))

Pearson's product-moment correlation

data:  standardise(longley$Unemployed) and standardise(longley$Armed.Forces)
t = -0.6745, df = 14, p-value = 0.5109
alternative hypothesis: true correlation is not equal to 0
95 percent confidence interval:
-0.6187113  0.3489766
sample estimates:
cor
-0.1774206


This hasn't reduced the correlation or therefore the albeit limited linear dependence of vectors.

What's going on?

R

-

## 3 Answers

It doesn't change the collinearity between the main effects at all. Scaling doesn't either. Any linear transform won't do that. What it changes is the correlation between main effects and their interactions. Even if A and B are independent with a correlation of 0, the correlation between A, and A:B will be dependent upon scale factors.

Try this...

a <- rnorm(50)
b <- rnorm(50)
cor(a,b)


This should be near 0.

cor(a, a*b)


This should also be near 0.

However...

b <- rnorm(50, 5, 2)
cor(a,b)


This will again have a tiny correlation... but!!

cor(a, a*b)


Now that should have a substantial correlation which you can make go away by centring and/or standardizing. I generally go with just the centring.

-
 +1 for comprehensive and comprehensible answer (with code!) – Peter Flom Oct 8 '11 at 19:51 It is also useful if you want to include, say, a quadratic term. – Aniko Oct 8 '11 at 21:47 absolutely Aniko – John Oct 9 '11 at 0:09 Best answer - thanks for this. I may have done the book an injustice in misinterpreting it too, but perhaps it was worth it to expose my ignorance. – rosser Oct 10 '11 at 11:19

Standardization does not affect the correlation between variables. They remain exactly the same. The correlation captures the synchronization of the direction of the variables. There is nothing in standardization that does change the direction of the variables.

If you want to eliminate multicollinearity between your variables, I suggest using Principal Component Analysis (PCA). As you know PCA is very effective in eliminating the multicollinearity problem. On the other hand PCA renders the combined variables (principal components P1, P2, etc...) rather opaque. A PCA model is always a lot more challenging to explain than a more traditional multivariate one.

-

Standardization is a common way to reduce collinearity. (You should be able to verify very quickly that it works by trying it out on a couple of pairs of variables.) Whether you do it routinely depends on how much of a problem collinearity is in your analyses.

Edit: I see I was in error. What standardizing does do, though, is reduce collinearity with product terms (interaction terms).

-
 Hmm, could you explain? Standardizing just changes the mean and variance of a random variable (to 0 and 1 respectively). This shouldn't change the correlation between two variables. I see how standardization can improve computational efficiency, but not how it reduces multicolinearity. – Charlie Oct 8 '11 at 16:15 No, I;m lost ... how can that possibly change the linear dependence of the column elements in the matrix of predictors. (Isn't that what collinearity is about?) – rosser Oct 8 '11 at 16:19 Although it is not correct that standardization changes collinearity in a purely mathematical sense, it can improve the numerical stability of algorithms to solve linear systems. That might be the source of the confusion in this reply. – whuber♦ Oct 8 '11 at 19:40 Standardization just does not reduce multicollinearity. It typically does not change the correlation between variables at all. – Gaetan Lion Oct 10 '11 at 17:10