14
$\begingroup$

I've come across a very good text on Bayes/MCMC. IT suggests that a standardisation of your independent variables will make an MCMC (Metropolis) algorithm more efficient, but also that it may reduce (multi)collinearity. Can that be true? Is this something I should be doing as standard.(Sorry).

Kruschke 2011, Doing Bayesian Data Analysis. (AP)

edit: for example

     > data(longley)
     > cor.test(longley$Unemployed, longley$Armed.Forces)

Pearson's product-moment correlation

     data:  longley$Unemployed and longley$Armed.Forces 
     t = -0.6745, df = 14, p-value = 0.5109
     alternative hypothesis: true correlation is not equal to 0 
     95 percent confidence interval:
     -0.6187113  0.3489766 
     sample estimates:
      cor 
     -0.1774206 

     > standardise <- function(x) {(x-mean(x))/sd(x)}
     > cor.test(standardise(longley$Unemployed), standardise(longley$Armed.Forces))

Pearson's product-moment correlation

     data:  standardise(longley$Unemployed) and standardise(longley$Armed.Forces) 
     t = -0.6745, df = 14, p-value = 0.5109
      alternative hypothesis: true correlation is not equal to 0 
     95 percent confidence interval:
      -0.6187113  0.3489766 
      sample estimates:
       cor 
     -0.1774206 

This hasn't reduced the correlation or therefore the albeit limited linear dependence of vectors.

What's going on?

R

$\endgroup$
22
$\begingroup$

It doesn't change the collinearity between the main effects at all. Scaling doesn't either. Any linear transform won't do that. What it changes is the correlation between main effects and their interactions. Even if A and B are independent with a correlation of 0, the correlation between A, and A:B will be dependent upon scale factors.

Try the following in an R console. Note that rnorm just generates random samples from a normal distribution with population values you set, in this case 50 samples. The scale function standardizes the sample to a mean of 0 and SD of 1.

set.seed(1) # the samples will be controlled by setting the seed - you can try others
a <- rnorm(50, mean = 0, sd = 1)
b <- rnorm(50, mean = 0, sd = 1)
mean(a); mean(b)
# [1] 0.1004483 # not the population mean, just a sample
# [1] 0.1173265
cor(a ,b)
# [1] -0.03908718

The incidental correlation is near 0 for these independent samples. Now normalize to mean of 0 and SD of 1.

a <- scale( a )
b <- scale( b )
cor(a, b)
# [1,] -0.03908718

Again, this is the exact same value even though the mean is 0 and SD = 1 for both a and b.

cor(a, a*b)
# [1,] -0.01038144

This is also very near 0. (a*b can be considered the interaction term)

However, usually the SD and mean of predictors differ quite a bit so let's change b. Instead of taking a new sample I'll rescale the original b to have a mean of 5 and SD of 2.

b <- b * 2 + 5
cor(a, b)
 # [1] -0.03908718

Again, that familiar correlation we've seen all along. The scaling is having no impact on the correlation between a and b. But!!

cor(a, a*b)
# [1,] 0.9290406

Now that will have a substantial correlation which you can make go away by centring and/or standardizing. I generally go with just the centring.

$\endgroup$
4
  • 1
    $\begingroup$ +1 for comprehensive and comprehensible answer (with code!) $\endgroup$ – Peter Flom Oct 8 '11 at 19:51
  • 1
    $\begingroup$ It is also useful if you want to include, say, a quadratic term. $\endgroup$ – Aniko Oct 8 '11 at 21:47
  • $\begingroup$ absolutely Aniko $\endgroup$ – John Oct 9 '11 at 0:09
  • 1
    $\begingroup$ Best answer - thanks for this. I may have done the book an injustice in misinterpreting it too, but perhaps it was worth it to expose my ignorance. $\endgroup$ – user6666 Oct 10 '11 at 11:19
7
$\begingroup$

As others have already mentioned, standardization has really nothing to do with collinearity.

Perfect collinearity

Let's start with what standardization (a.k.a. normalization) is, what we mean by it is subtracting the mean and dividing by the standard deviation so that the resulting mean is equal to zero and standard deviation to unity. So if random variable $X$ has mean $\mu_X$ and standard deviation $\sigma_X$, then

$$ \newcommand{Var}{\mathrm{Var}} Z_X = \frac{X - \mu_X}{\sigma_X} $$

has mean $\mu_Z = 0$ and standard deviation $\sigma_Z = 1$ given the properties of expected value and variance that $E(X + a) = E(X) + a$, $E(bX) = b\,E(X)$ and $\Var(X + a) = \Var(X)$, $\Var(bX) = b^2 \Var(X)$, where $X$ is r.v. and $a,b$ are constants.

We say that two variables $X$ and $Y$ are perfectly collinear if there exists such values $\lambda_0$ and $\lambda_1$ that

$$ Y = \lambda_0 + \lambda_1 X $$

what follows, if $X$ has mean $\mu_X$ and standard deviation $\sigma_X$, then $Y$ has mean $\mu_Y = \lambda_0 + \lambda_1 \mu_X$ and standard deviation $\sigma_Y = \lambda_1 \sigma_X$. Now, when we standardize both variables (remove their means and divide by standard deviations), we get $Z_X = Z_X$...

Correlation

Of course perfect collinearity is not something that we would see that often, but strongly correlated variables may also be a problem (and they are related species with collinearity). So does the standardization affect correlation? Please compare the following plots showing two correlated variables on two plots before and after scaling: enter image description here

Can you spot the difference? As you can see, I purposefully removed the axis labels, so to convince you that I'm not cheating, see the plots with added labels:

enter image description here

Mathematically speaking, if correlation is

$$ \newcommand{Corr}{\mathrm{Corr}} \newcommand{Cov}{\mathrm{Cov}} \Corr(X, Y) = \frac{\Cov(X,Y)}{\Var(X)\,\Var(Y)} $$

then with collinear variables we have

$$ \require{cancel} \begin{align} \Corr(X, Y) &= \frac{E[(X - \mu_X)(Y - \mu_Y)]}{\sigma_X\sigma_Y} \\ &=\frac{E[(X - \mu_X)(\cancel{\lambda_0} + \lambda_1 X - \cancel{\lambda_0} - \lambda_1\mu_X )]}{\sigma_X\;\lambda_1\sigma_X} \\ &= \frac{E[(X - \mu_X)(\lambda_1 X - \lambda_1\mu_X )]}{\sigma_X\;\lambda_1\sigma_X} \\ &= \frac{E[(X - \mu_X)\lambda_1( X - \mu_X )]}{\sigma_X\;\lambda_1\sigma_X} \\ &= \frac{\cancel{\lambda_1} E[(X - \mu_X)(X - \mu_X )]}{\sigma_X\;\cancel{\lambda_1}\sigma_X} \\ &= \frac{E[(X - \mu_X)(X - \mu_X )]}{\sigma_X\sigma_X} \end{align} $$

now since $\Cov(X,X) = \Var(X)$,

$$ \begin{align} &= \frac{\Cov(X, X)}{\sigma_X^2} = \frac{\Var(X)}{\Var(X)} = 1 \end{align} $$

While with standardized variables

$$ \begin{align} \Corr(Z_X, Z_Y) &= \frac{E[(Z_X - 0)(Z_Y - 0)]}{1 \times 1} \\ &= \Cov(Z_X, Z_Y) = \Var(Z_X) = 1 \end{align} $$

since $Z_X = Z_Y$...

Finally, notice that what Kruschke is talking about, is that standardizing of the variables makes life easier for the Gibbs sampler and leads to reducing of correlation between intercept and slope in the regression model he presents. He doesn't say that standardizing variables reduces collinearity between the variables.

$\endgroup$
0
$\begingroup$

Standardization does not affect the correlation between variables. They remain exactly the same. The correlation captures the synchronization of the direction of the variables. There is nothing in standardization that does change the direction of the variables.

If you want to eliminate multicollinearity between your variables, I suggest using Principal Component Analysis (PCA). As you know PCA is very effective in eliminating the multicollinearity problem. On the other hand PCA renders the combined variables (principal components P1, P2, etc...) rather opaque. A PCA model is always a lot more challenging to explain than a more traditional multivariate one.

$\endgroup$
2
  • $\begingroup$ A modern alternative, often better, is regularization. $\endgroup$ – kjetil b halvorsen Jul 16 '19 at 10:02
  • $\begingroup$ I have tested variable selection between standard stepwise algorithms and LASSO. And, LASSO comes in a very distant second. LASSO penalizes variable influences, it can select weak variables over stronger variables. It can even cause variables signs to change. And, it breaks down the entire framework of statistical significance, Confidence Intervals, and Prediction Intervals. LASSO can at times work. But, look very carefully at the MSEs vs. Lambda graph and the Coefficients vs. Lambda graphs. That's where you can visually observe if your LASSO model worked. $\endgroup$ – Sympa Jul 19 '19 at 4:55
0
$\begingroup$

It doesn't reduce the collinearity, it can reduce the VIF. Commonly we use VIF as indicator for concerns for collinearity.

Source: http://blog.minitab.com/blog/adventures-in-statistics-2/what-are-the-effects-of-multicollinearity-and-when-can-i-ignore-them

$\endgroup$
1
  • 2
    $\begingroup$ Welcome to the site. At present this is more of a comment than an answer. You could expand it, perhaps by giving a summary of the information at the link, or we can convert it into a comment for you. In addition, my reading of the linked post isn't quite that standardizing reduces the VIF without reducing collinearity. Their example is very specific & more nuanced than that. $\endgroup$ – gung - Reinstate Monica Sep 19 '18 at 1:16
-3
$\begingroup$

Standardization is a common way to reduce collinearity. (You should be able to verify very quickly that it works by trying it out on a couple of pairs of variables.) Whether you do it routinely depends on how much of a problem collinearity is in your analyses.

Edit: I see I was in error. What standardizing does do, though, is reduce collinearity with product terms (interaction terms).

$\endgroup$
4
  • $\begingroup$ Hmm, could you explain? Standardizing just changes the mean and variance of a random variable (to 0 and 1 respectively). This shouldn't change the correlation between two variables. I see how standardization can improve computational efficiency, but not how it reduces multicolinearity. $\endgroup$ – Charlie Oct 8 '11 at 16:15
  • $\begingroup$ No, I;m lost ... how can that possibly change the linear dependence of the column elements in the matrix of predictors. (Isn't that what collinearity is about?) $\endgroup$ – user6666 Oct 8 '11 at 16:19
  • $\begingroup$ Although it is not correct that standardization changes collinearity in a purely mathematical sense, it can improve the numerical stability of algorithms to solve linear systems. That might be the source of the confusion in this reply. $\endgroup$ – whuber Oct 8 '11 at 19:40
  • $\begingroup$ Standardization just does not reduce multicollinearity. It typically does not change the correlation between variables at all. $\endgroup$ – Sympa Oct 10 '11 at 17:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy