# Is there a result showing a relation between the size of the residuals and the correlation coefficient?

Consider an OLS regression between two variables. Is there any result which relates the size of the residuals (measured, perhaps, by the sum of the squares) to the Pearson correlation coefficient of the two variables?

Informally, I would expect tightly correlated variables to produce small overall residuals and loosely correlated variables to produce large residuals.

-

Yes. $r^2= 1-SSE/SST$ whe $r$ is the correlation coefficient, $SSE$ is the error sum of squares, and $SST=\sum(y_i -\bar y)^2$ is the total sum of squares. In multiple regression, this is called "R square" but in simple linear regression, it really is just the square of the correlation coefficient.

-
Nice, that's pretty much what I had in mind. I'll allow for some time for other answers, just in case, then I'll accept an answer. Btw, could you outline the proof for this? Is it straightforward algebra? – Henry Henrinson Aug 23 '14 at 17:12

As per your request a sketch of a possible proof:

Note: the definition of SSR and SSE that I use is different from that of the other answer (I think his SSR is my SSE and vice versa). Both notations are used.

Consider the regression line $y_i=ax_i+b+e_i \quad i=1,\dots n$

Since we are looking at regression with intercept, we know that the regression line must go through the mass centre, the point ($\bar{x}, \bar{y}$). We then obtain that $y_i - \bar{y}=b(x_i-\bar{x}) + e_i$

This last quantity is the difference from the mean. The squared difference from the mean can then be decomposed into two parts: an unexplained and an explained part:

$\sum\limits_{i=1}^n (y_i - \bar{y})^2=b^2\sum\limits_{i=1}^n(x_i-\bar{x})^2 + \sum\limits_{i=1}^n e_i^2 \tag{*}$

Phrased differently: Sum of Squared Totals (SST) = Sum of Squared Residuals (SSR) + Sum of Squares Explained (SSE)

We then define $R^2$ (the coefficient of determination) to be:

$R^2=\frac{SSE}{SST}=\frac{b^2\sum\limits_{i=1}^n(x_i-\bar{x})^2}{\sum\limits_{i=1}^n (y_i - \bar{y})^2}$. If you substitute the formula of $b$ (the OLS estimator) in now, then it follows that $R^2= \frac{(\sum(x_i-\bar{x})(y_i-\bar{y}))^2}{\sum(x_i-\bar{x})^2\sum(y_i-\bar{y})^2}$. This makes $R$ equal to precisely the Pearson correlation coefficient.

It then follows from (*) that $R^2=1- \frac{\sum e_i^2}{\sum(y_i-\bar{y})^2}=1-\frac{SSR}{SST}$

Note that I used that there is an intercept in the regression model. Else this formula does not hold true!

-
Awesome, finally got a chance to go through the steps here and make it clear to myself why this is. I've also added some edits to clarify things for future reference. – Henry Henrinson Aug 26 '14 at 9:07
Glad I could be of help :) – rbm Aug 26 '14 at 9:09