1
$\begingroup$

I am a little bit confused with Collinearity and Correlation. As far as I know Collinearity (which is linear dependency between variables) implies a Correlation between the variables but the reverse is not true. Why so? If we have a strong correlation does it not imply we have a linear relationship between the variables?

$\endgroup$
  • $\begingroup$ You might like "Can You Actually TEST for Multicollinearity?" by Dave Giles. $\endgroup$ – Richard Hardy Apr 20 '18 at 18:07
  • $\begingroup$ In what sense do you mean "the reverse is not true"? A nonzero correlation is a form of multicollinearity--perhaps the simplest form there is. $\endgroup$ – whuber Apr 20 '18 at 18:08
0
$\begingroup$

Nonlinear relationship can have linear correlation. That is why linear correlation doesn't imply linear relationship, the one that is called collinearity.

Look at two cases:

  1. $X_1=a+bX_2$
  2. $X_1=e^{X_2}$

Where $X\sim Uniform(0,1)$

In both cases, you'll have correlation $Cov(X_1,X_2)\ne 0$, but only the first case leads to multicollinearity, a technical issue of identification due to rank deficiency of $X'X$ matrix.

In OLS you have the following estimate of the parameters: $$\hat\beta=(X'X)^{-1}yX$$ In case of perfect collinearity you get a singular matrix $(X'X)^{-1}$. You can think of a first case as where $X_1$ is a temperature in Celsius while $X_2$ is the temperature in Fahrenheit. Obviously, it doesn't make a sense to have both variables in the model.

$\endgroup$
  • 1
    $\begingroup$ I guess your answer is fine, but when talking of linear relationships, I am tempted to think about linear models such as $y=\beta_0+\beta_1 x+\varepsilon$ where there is some noise and the relationship is not perfect. Should I just resist the temptation and forget the linear model or would you include some note on that in the answer? $\endgroup$ – Richard Hardy Apr 20 '18 at 17:57
  • $\begingroup$ @RichardHardy, do you want me to distinguish multicollinearity and perfect multicollinearity, with the latter bringing identification issue? do you mean something else? $\endgroup$ – Aksakal Apr 20 '18 at 18:07
  • $\begingroup$ I just thought that you give an example of a linear relationship without an error term while in statistics, models including error terms are much more prevalent. Your answer could be more relevant if you included such an example in addition to, if not in place of, the one without the error. And perhaps mention the difference or definitions of perfect vs. imperfect multicollinearity. $\endgroup$ – Richard Hardy Apr 20 '18 at 18:09
  • $\begingroup$ @RichardHardy, actually I meant to present an ideal case of a perfect collinearity, e.g. a model with two variables for temperatures in Celsius and Fahrenheit scales. This is to simplify the exposition and get to the root of the confusion $\endgroup$ – Aksakal Apr 20 '18 at 18:13
0
$\begingroup$

Let us say you have a dependent variable $Y$ built as a linear combination of independent variables $Y$ = $\Sigma_i^N \alpha_i X_i$. You are in a situation of multicollinearity if there is an ind. variable $X_j$ that can be written as a linear combination of $X_j = \Sigma_{i\neq j}^N \beta_i X_i$; in plain english, one of the supposedly independent variables can be written as a linear combination of (some of) the others.

In this nomenclature, if you only have one index $\beta_i$ (let us say, $h$) that is significantly different from 0, while the others are very similar to 0, then you will be able to measure a very strong correlation between $X_j$ and $X_h$. However, this might not be as easily measurable when you have several $\beta_i \neq 0$ (and yet you are in a situation of multicollinearity).

As a side remark, if you have a correlation between two variables X and Y, you do not have multicollinearity.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.