14
$\begingroup$

In linear models we need to check if a relationship exists among the explanatory variables. If they correlate too much then there is collinearity (i.e., the variables partly explain each other). I am currently just looking at the pairwise correlation between each of the explanatory variables.

Question 1: What classifies as too much correlation? For example, is a Pearson correlation of 0.5 too much?

Question 2: Can we fully determine whether there is collinearity between two variables based on the correlation coefficient or does it depend on other factors?

Question 3: Does a graphical check of the scatterplot of the two variables add anything to what the correlation coefficient indicates?

$\endgroup$
  • 2
    $\begingroup$ Collinearity (singularity) among 3+ variables is not reduced only to high pairwise correlations. Search the site for questions tagged "multicollinearity". Also, I recommend you to read this my answer: stats.stackexchange.com/a/70910/3277. $\endgroup$ – ttnphns May 27 '14 at 13:30
15
$\begingroup$
  1. There is no 'bright line' between not too much collinearity and too much collinearity (except in the trivial sense that $r = 1.0$ is definitely too much). Analysts would not typically think of $r = .50$ as too much collinearity between two variables. A rule of thumb regarding multicollinearity is that you have too much when the VIF is greater than 10 (this is probably because we have 10 fingers, so take such rules of thumb for what they're worth). The implication would be that you have too much collinearity between two variables if $r \ge .95$. You can read more about the VIF and multicollinearity in my answer here: What is the effect of having correlated predictors in a multiple regression model?

  2. This depends on what you mean by "fully determine". If the correlation between two variables were $r \ge .95$, then most data analysts would say you had problematic collinearity. However, you can have multiple variables where no two variables have a pairwise correlation that high, and still have problematic collinearity hidden amongst the whole set of variables. This is where other metrics, such as the VIFs and condition numbers come in handy. You can read more on this topic at my question here: Is there a reason to prefer a specific measure of multicollinearity?

  3. It is always smart to look at your data, and not simply numerical summaries / test results. The canonical reference here is Anscomb's quartet.

$\endgroup$
3
$\begingroup$

My take on the three questions is

Question 1 What classifies as too much correlation? For example: a pearson correlation of 0.5 is that too much?

Many authors argue that (multi-)collinearity is not a problem. Take a look here and here for a rather acid opinion on the subject. The bottom line is that multicollinearity does not have an impact on the hypothesis testing other than having a lower (effective) sample size. It will be hard for you to interpret the regression coefficients if you do a regression, for instance, but you do not violate any basic assumption if you choose to do so.

Question 2 Can we fully determine whether there is collinearity between two variables based on the correlation coefficient or does it depend on other factors?

I think there are several ways of measuring the correlation between two variables, from calculating Pearson's correlation coefficient (if you assume linearity, and apparently you did so), to Spearman's rank, distance correlation, and even doing PCA on your dataset. But I would leave the answer of this question to better informed people than me.

Question 3 Does a graphical check of the scatter plot of the two variables add anything to what the correlation coefficient indicates?

IMO, the answer is sound no.

$\endgroup$
  • 2
    $\begingroup$ IMHO, the answer to (3) is on the contrary a very strong yes: whereas the correlation coefficient can give only a single numerical assessment of the linearity of a relationship, a quick glance at the scatterplot will provide a wealth of additional information about that relationship, including behaviors that were not expected beforehand. However, the real interest in this set of questions lies in how to assess relationships among three or more variables (despite how (3) was actually phrased), and in that case even a scatterplot matrix does not reveal everything, as @ttnphns notes. $\endgroup$ – whuber May 27 '14 at 15:54
  • 1
    $\begingroup$ As far as (1) goes, I read your reference (to Dave Gile's blog) differently: he argues that formal testing of multicollinearity is misguided. I do not see him claiming that multicollinearity is not a problem. $\endgroup$ – whuber May 27 '14 at 15:57
  • $\begingroup$ My understanding of Dave Gile's answer is that the only way multicollinearity impacts the results will be through an equivalent smaller sample size. So just like it makes no sense to test for small sample size, it makes no sense to test the impact of multicollinearity. But I would be happy to hear your opinion on it, maybe I misunderstood it. $\endgroup$ – pedrofigueira May 27 '14 at 16:07
  • $\begingroup$ Well, needing a larger sample size can be a huge impact for most studies! A subtler effect of near-collinearity concerns model building and variable selection, as discussed (inter alia) in threads such as stats.stackexchange.com/questions/50537 and stats.stackexchange.com/a/28476/919. But let's make sure we're talking about the same things: Giles is discussing formal tests of multicollinearity, as if the independent variables were randomly sampled. Here the concern seems focused on using multicollinearity diagnostics to understand the capabilities and limitations of a model. $\endgroup$ – whuber May 27 '14 at 16:17
1
$\begingroup$

A common way to evaluate collinearity is with variance inflation factors (VIFs). This can be achieved in R using the 'vif' function within the 'car' package. This has an advantage over looking at only the correlations between two variables, as it simultaneously evaluates the correlation between one variable and the rest of the variables in the model. It then gives you a single score for each predictor in the model.

As stated above there is no hard and fast cutoff, but VIF scores are often decided to be problematic once they are between 5-10. I use field specific rules of thumb for this. Also- there is nothing necessarily invalid about using correlated predictors (so long as they are not perfectly correlated). You will just need more data to separate effects. When you don't have enough data there will be large uncertainties in the parameter estimates of the correlated predictors, and these estimates will be sensitive to re-sampling.

To answer your questions specifically:

  1. Don't use correlation coefficients. use VIFs of the model with all predictors and no interactions. VIFs of 5-10 are indicating too much correlation, your specific cutoff depends on what you need to do with the model.

  2. It depends on the other predictors in the model, which is why it is beneficial to use VIFs.

  3. Nope! The statistics will better quantify what you are eyeballing with the scatter plot. Unless there is a super violation of the assumptions of OLS when regressing your predictors against each other.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.