Correlation and collinearity in regression

Question

I did a correlation analysis for my variables. All of them are associated (the coefficient is above 0). However, there is no collinearity problem in my regression analysis. I do not know how to explain it? I think I am confused about correlation and collinearity analysis...but if coefficient is above 0, it shows there is an association, why no collinearity problem shown in regression analysis? By the way, what is the difference between regression analysis and correlation analysis?

Answer 1

In a linear regression context, we designate one variable as the "dependent" one, and the rest as "explanatory" or "regressors". Then:

As @ssdecontrol answer noted, in order for the regression to give good results we would want that the dependent variable is correlated with the regressors -since the linear regression does exactly that -it attempts to quantify the correlation (understood in general terms) of the dependent variable with the regressors.

Regarding the interrelation between the regressors: if they have zero-correlation, then running a multiple linear regression provides the same coefficient estimates as running many simple regressions (i.e. regress the dependent variable on each regressor separately). So the usefulness of multiple linear regression emerges when the regressors are correlated between them...

...but what about colinearity? Well, I suggest you start to call it "perfect collinearity" and "near-perfect colinearity" -because it is in such cases that the estimation algorithm breaks down. It is the case when the data series of a regressor can be exactly, or almost exactly, written as a linear combination of other regressors. This will make the regressor matrix singular or near-singular and so problematic to invert, as is needed for the least-square solution.

In sum, we want the regressors to exhibit among them some degree of colinearity (in order to justify using multiple regression), but we do not want them to be perfectly or near perfectly colinear.

Answer 2

Correlation is necessary but not sufficient to cause collinearity.

Correlation is a measure of the strength of linear association between to variables. That is, high correlation between $X$ and $Y$ means that the relationship between them is very close to $aX + b = Y$ where $a$ and $b$ are some constants.

Regression is a technique for estimating what those constants might be, under the assumption that the true relationship is linear. It finds the line that minimizes the distance from the line, in the $(x,y)$ space, to every observed data point. The relationship between regression and correlation is that regression is a good fit when correlation is high. This should make sense in light of the definition for correlation I gave above.