2
$\begingroup$

I'm taking an intro to statistics class and currently we are covering linear regression. The course is good but unfortunately it doesn't always gives all the information that is needed. For example it gives formulas for the slope and y-intercept as

$b=(r)\frac{s_Y}{s_X}=\frac{n \sum{xy} - \sum{x} \sum{y}}{n \sum{x^2} - (\sum{x})^2}$ , where
$r$ is the correlation coefficient
$s_Y$ is the standard deviation of the $Y$ scores
$s_X$ is the standard deviation of the $X$ scores

and

$a = \bar{y} - b \bar{x} = \frac{\sum{y} - b \sum{x}}{n}$

However it doesn't explain why the formulas are what they are. In other words it doesn't explain how these formulas where derived. Can someone show the derivation, especially the one for the slope, $b$, i.e. why $b=(r) \frac{s_Y}{s_X}$?

Any help is appreciated.

$\endgroup$
  • $\begingroup$ The estimates given are the least squares estimates, which means they give us a line that minimizes the sum of the squared vertical distances between the line and the actual points. In particular, we minimize the function $\sum_{i=1}^{n}(y_i - \beta_0 - \beta_1 x_i)^2$ with respect to $\beta_0$ and $\beta_1$. The derivation itself is usually carried out using calculus. $\endgroup$ – dsaxton Mar 2 '16 at 17:36
  • 1
    $\begingroup$ See stats.stackexchange.com/search?q=regression+formula+derivation . A method that involves no Calculus, is very general, and obtains these formulas as a special case, is presented in my answer at stats.stackexchange.com/a/71303. The only knowledge beyond elementary algebra that it presumes is purely geometrical: how to visualize linear transformations of plane figures. $\endgroup$ – whuber Mar 2 '16 at 17:41
4
$\begingroup$

The ideal line for the population is:

$\large \mu_i\,=\,\beta_o \,+\,\beta_1\,X_i$


enter image description here

The estimated regression line is:

$\large \hat \mu_i\,=\,\hat\beta_o\,+\,\hat\beta_1\,X_i$

We want to minimize the squared distances from all the observed values $Y_i$ in the population and $\mu_i$, which are the fitted values that we would get if we had the “ideal” regression line calculated based on all the population - it has the symbol of mean, because it’s the mean of every bell’s curve as in the diagram above; $\hat \mu_i$ is the fitted values for the sample based on our estimated regression line.

Just subtracting and adding $\hat \mu_i$:

$\large \displaystyle \sum_{i=1}^n (Y_i-\mu_i)^2 = \displaystyle \sum_{i=1}^{n}(Y_i-\hat \mu_i + \hat \mu_i - \mu_i)^2= \displaystyle \sum_{i=1}^{n}(Y_i-\hat \mu_i)^2 + 2 \sum_{i=1}^{n} (Y_i - \hat \mu_i)\,(\hat \mu_i - \mu_i) + \displaystyle \sum_{i=1}^n (\hat \mu_i - \mu_i)^2$

Finding the minimum amounts to:

$\Large\sum_{i=1}^{n} (Y_i - \hat \mu_i)\,(\hat \mu_i - \mu_i)\,=\,0\,\small \tag 1$


... leaving us with:


$\displaystyle \sum_{i=1}^{n}(Y_i-\mu_i)^2 = \displaystyle \sum_{i=1}^{n}(Y_i-\hat \mu_i)^2 + \displaystyle \sum_{i=1}^n (\hat \mu_i - \mu_i)^2$


It follows that:


$\displaystyle \sum_{i=1}^{n}(Y_i-\mu_i)^2\,\geq \sum_{i=1}^{n}(Y_i-\hat \mu_i)^2$

Considering only horizontal lines, and from equation (1):

$\large \displaystyle\sum_{i=1}^{n} (Y_i - \hat \beta_o)\,(\hat \beta_o - \beta_o)\,=\,(\hat \beta_o - \beta_o)\,\displaystyle\sum_{i=1}^{n}(Y_i - \hat \beta_o)=\,0$


This will happen if:


$\displaystyle\sum_{i=1}^{n}(Y_i - \hat \beta_o)=\,0$. In other words if $n\,\bar Y\,-\,n\,\hat \beta_o\,=\,0$, or



$\Large \bar Y = \beta_o \small \tag 2$.

If we do regression through the origin:


$\displaystyle\sum_{i=1}^{n} (Y_i - \hat \mu_i)\,(\hat \mu_i - \mu_i)\,=\,\displaystyle\sum_{i=1}^{n} (Y_i - \hat \beta_1\, X_i)\,(\hat \beta_1 X_i - \beta_1\,X_i)\,=\,\displaystyle\sum_{i=1}^{n}(Y_i\,\hat \beta_1\,X_i\,-\,Y_i\beta_1 X_i)(-\hat \beta_1 X_i \hat \beta_1 X_i\, +\,\hat\beta_1 X_i \beta_1 X_i)$

$=\,(\hat \beta_1 - \beta_1)\,\displaystyle\sum_{i=1}^{n}(Y_iX_i)-(\hat \beta_1 X_i^2)$. And this will be zero if:

$\displaystyle\sum_{i=1}^{n}(Y_iX_i)-(\hat \beta_1 X_i^2)=\displaystyle\sum_{i=1}^{n}(Y_iX_i)-\hat \beta_1\displaystyle\sum_{i=1}^{n}X_i^2=0$. Hence,

$\Large \hat \beta_1=\frac{\displaystyle\sum_{i=1}^{n}Y_iX_i}{\displaystyle\sum_{i=1}^{n}X_i^2} \small \tag 3$.

Now doing both intercept and slope:

$0 =\displaystyle\sum_{i=1}^{n} (Y_i - \hat \mu_i)\,(\hat \mu_i - \mu_i)\, =\displaystyle\sum_{i=1}^{n}(Y_i -\hat\beta_o - \hat \beta_1 X_i)(\hat\beta_o+\hat\beta_1X_i-\beta_o-\beta_1X_i)$

$= \displaystyle\sum_{i=1}^{n} (Y_i -\hat\beta_o \hat \beta_1 X_i)((\hat\beta_o-\beta_o)+\hat\beta_1X_i-\beta_1X_i)$

$=\large (\hat \beta_o-\beta_o)\displaystyle\sum_{i=1}^{n}(Y_i-\hat\beta_o-\hat\beta_1X_i)+(\hat\beta_1-\beta_1)\displaystyle\sum_{i=1}^{n}(Y_i-\hat\beta_o-\hat\beta_1X_i)X_i \small \tag 4$



$0 = \displaystyle\sum_{i=1}^{n}(Y_i-\hat\beta_o-\hat\beta_1X_i)=n\bar Y_i-n\hat\beta_o-n\hat\beta_1\bar X_i$. Hence,

$\Large \hat\beta_o\,=\,\bar Y\,-\,\hat\beta_1\,\bar X \small \tag 5$



And for the second part of equation (4):

$0=\displaystyle\sum_{i=1}^{n}(Y_i-\hat\beta_o-\hat\beta_1X_i)X_i$. Substituting $\hat\beta_o=\bar Y- \hat \beta_1 \bar X$ for $\hat\beta_o$:



$0=\displaystyle\sum_{i=1}^{n}(Y_i-\bar Y + \hat \beta_1 \bar X - \hat \beta_1X_i)X_i =\displaystyle\sum_{i=1}^{n}(Y_i-\bar Y)X_i - \hat\beta_1 \sum_{i=1}^{n}(X_i-\bar X)X_i$



Since $\displaystyle\sum_{i=1}^{n}(Y_i-\bar Y)=0$, also $\displaystyle\bar X\sum_{i=1}^{n}(Y_i-\bar Y)=0$.



$\large\hat \beta_1\,=\,\frac{\sum_{i=1}^{n}(Y_i-\bar Y)X_i}{\sum_{i=1}^{n}(X_i-\bar X)X_i} =\frac{\sum_{i=1}^{n}(Y_i-\bar Y)(X_i-\bar X)}{\sum_{i=1}^{n}(X_i-\bar X)(X_i-\bar X)}$



Checking equation (1):

$\large \frac{\sum_{i=1}^{n}(Y_i-\bar Y)(X_i-\bar X)}{\sum_{i=1}^{n}(X_i-\bar X)(X_i-\bar X)} =\frac{cov(X,Y)}{SD(X)}=\frac{\frac{cor(X,Y)}{SD(X)SD(Y)}}{SD(X)}$. Hence,



$\Large \hat\beta_1\,=\,cor(Y, X)\,\frac{SD(Y)}{SD(X)}\,=\,\frac{cov(Y,X)}{var(X)}\small \tag 6$

$\endgroup$
  • $\begingroup$ This is from my notes on statistics, where I have the derivation with and without linear algebra. $\endgroup$ – Antoni Parellada Mar 2 '16 at 18:51
  • $\begingroup$ You might enjoy the challenge of obtaining equation (6) with these same algebraic ideas but with a minimum of algebraic manipulation. I believe it should be possible to do in a few lines rather than in many pages. Such efforts tend to be rewarded by additional insight. Or, you might choose to do it in one line using Euclidean geometry. $\endgroup$ – whuber Mar 2 '16 at 18:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.