# Does standardising independent variables reduce collinearity?

I've come across a very good text on Bayes/MCMC. IT suggests that a standardisation of your independent variables will make an MCMC (Metropolis) algorithm more efficient, but also that it may reduce (multi)collinearity. Can that be true? Is this something I should be doing as standard.(Sorry).

Kruschke 2011, Doing Bayesian Data Analysis. (AP)

R

edit: for example

     > data(longley)
> cor.test(longley$Unemployed, longley$Armed.Forces)

Pearson's product-moment correlation

data:  longley$Unemployed and longley$Armed.Forces
t = -0.6745, df = 14, p-value = 0.5109
alternative hypothesis: true correlation is not equal to 0
95 percent confidence interval:
-0.6187113  0.3489766
sample estimates:
cor
-0.1774206

> standardise <- function(x) {(x-mean(x))/sd(x)}
> cor.test(standardise(longley$Unemployed), standardise(longley$Armed.Forces))

Pearson's product-moment correlation

data:  standardise(longley$Unemployed) and standardise(longley$Armed.Forces)
t = -0.6745, df = 14, p-value = 0.5109
alternative hypothesis: true correlation is not equal to 0
95 percent confidence interval:
-0.6187113  0.3489766
sample estimates:
cor
-0.1774206


This hasn't reduced the correlation or therefore the albeit limited linear dependence of vectors.

What's going on?

R

-

It doesn't change the collinearity between the main effects at all. Scaling doesn't either. Any linear transform won't do that. What it changes is the correlation between main effects and their interactions. Even if A and B are independent with a correlation of 0, the correlation between A, and A:B will be dependent upon scale factors.

Try this...

a <- rnorm(50)
b <- rnorm(50)
cor(a,b)


This should be near 0.

cor(a, a*b)


This should also be near 0.

However...

b <- rnorm(50, 5, 2)
cor(a,b)


This will again have a tiny correlation... but!!

cor(a, a*b)


Now that should have a substantial correlation which you can make go away by centring and/or standardizing. I generally go with just the centring.

-
+1 for comprehensive and comprehensible answer (with code!) –  Peter Flom Oct 8 '11 at 19:51
It is also useful if you want to include, say, a quadratic term. –  Aniko Oct 8 '11 at 21:47
absolutely Aniko –  John Oct 9 '11 at 0:09
Best answer - thanks for this. I may have done the book an injustice in misinterpreting it too, but perhaps it was worth it to expose my ignorance. –  rosser Oct 10 '11 at 11:19

Standardization does not affect the correlation between variables. They remain exactly the same. The correlation captures the synchronization of the direction of the variables. There is nothing in standardization that does change the direction of the variables.

If you want to eliminate multicollinearity between your variables, I suggest using Principal Component Analysis (PCA). As you know PCA is very effective in eliminating the multicollinearity problem. On the other hand PCA renders the combined variables (principal components P1, P2, etc...) rather opaque. A PCA model is always a lot more challenging to explain than a more traditional multivariate one.

-