Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

Consider an OLS regression between two variables. Is there any result which relates the size of the residuals (measured, perhaps, by the sum of the squares) to the Pearson correlation coefficient of the two variables?

Informally, I would expect tightly correlated variables to produce small overall residuals and loosely correlated variables to produce large residuals.

share|improve this question

2 Answers 2

up vote 6 down vote accepted

Yes. $r^2= 1-SSE/SST$ whe $r$ is the correlation coefficient, $SSE$ is the error sum of squares, and $SST=\sum(y_i -\bar y)^2$ is the total sum of squares. In multiple regression, this is called "R square" but in simple linear regression, it really is just the square of the correlation coefficient.

share|improve this answer
    
Nice, that's pretty much what I had in mind. I'll allow for some time for other answers, just in case, then I'll accept an answer. Btw, could you outline the proof for this? Is it straightforward algebra? –  Henry Henrinson Aug 23 at 17:12

As per your request a sketch of a possible proof:

Note: the definition of SSR and SSE that I use is different from that of the other answer (I think his SSR is my SSE and vice versa). Both notations are used.

Consider the regression line $y_i=ax_i+b+e_i \quad i=1,\dots n$

Since we are looking at regression with intercept, we know that the regression line must go through the mass centre, the point ($\bar{x}, \bar{y}$). We then obtain that $y_i - \bar{y}=b(x_i-\bar{x}) + e_i$

This last quantity is the difference from the mean. The squared difference from the mean can then be decomposed into two parts: an unexplained and an explained part:

$\sum\limits_{i=1}^n (y_i - \bar{y})^2=b^2\sum\limits_{i=1}^n(x_i-\bar{x})^2 + \sum\limits_{i=1}^n e_i^2 \tag{*}$

Phrased differently: Sum of Squared Totals (SST) = Sum of Squared Residuals (SSR) + Sum of Squares Explained (SSE)

We then define $R^2$ (the coefficient of determination) to be:

$R^2=\frac{SSE}{SST}=\frac{b^2\sum\limits_{i=1}^n(x_i-\bar{x})^2}{\sum\limits_{i=1}^n (y_i - \bar{y})^2}$. If you substitute the formula of $b$ (the OLS estimator) in now, then it follows that $R^2= \frac{(\sum(x_i-\bar{x})(y_i-\bar{y}))^2}{\sum(x_i-\bar{x})^2\sum(y_i-\bar{y})^2}$. This makes $R$ equal to precisely the Pearson correlation coefficient.

It then follows from (*) that $R^2=1- \frac{\sum e_i^2}{\sum(y_i-\bar{y})^2}=1-\frac{SSR}{SST}$

Note that I used that there is an intercept in the regression model. Else this formula does not hold true!

share|improve this answer
    
Awesome, finally got a chance to go through the steps here and make it clear to myself why this is. I've also added some edits to clarify things for future reference. –  Henry Henrinson Aug 26 at 9:07
    
Glad I could be of help :) –  rbm Aug 26 at 9:09

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.