slope and intercept of linear regression

Question

I'm taking an intro to statistics class and currently we are covering linear regression. The course is good but unfortunately it doesn't always gives all the information that is needed. For example it gives formulas for the slope and y-intercept as

$b=(r)\frac{s_Y}{s_X}=\frac{n \sum{xy} - \sum{x} \sum{y}}{n \sum{x^2} - (\sum{x})^2}$ , where
$r$ is the correlation coefficient
$s_Y$ is the standard deviation of the $Y$ scores
$s_X$ is the standard deviation of the $X$ scores

and

$a = \bar{y} - b \bar{x} = \frac{\sum{y} - b \sum{x}}{n}$

However it doesn't explain why the formulas are what they are. In other words it doesn't explain how these formulas where derived. Can someone show the derivation, especially the one for the slope, $b$, i.e. why $b=(r) \frac{s_Y}{s_X}$?

Any help is appreciated.

Answer 1

The ideal line for the population is:

$\large \mu_i\,=\,\beta_o \,+\,\beta_1\,X_i$

The estimated regression line is:

$\large \hat \mu_i\,=\,\hat\beta_o\,+\,\hat\beta_1\,X_i$

We want to minimize the squared distances from all the observed values $Y_i$ in the population and $\mu_i$, which are the fitted values that we would get if we had the “ideal” regression line calculated based on all the population - it has the symbol of mean, because it’s the mean of every bell’s curve as in the diagram above; $\hat \mu_i$ is the fitted values for the sample based on our estimated regression line.

Just subtracting and adding $\hat \mu_i$:

$\large \displaystyle \sum_{i=1}^n (Y_i-\mu_i)^2 = \displaystyle \sum_{i=1}^{n}(Y_i-\hat \mu_i + \hat \mu_i - \mu_i)^2= \displaystyle \sum_{i=1}^{n}(Y_i-\hat \mu_i)^2 + 2 \sum_{i=1}^{n} (Y_i - \hat \mu_i)\,(\hat \mu_i - \mu_i) + \displaystyle \sum_{i=1}^n (\hat \mu_i - \mu_i)^2$

Finding the minimum amounts to:

$\Large\sum_{i=1}^{n} (Y_i - \hat \mu_i)\,(\hat \mu_i - \mu_i)\,=\,0\,\small \tag 1$

... leaving us with:

$\displaystyle \sum_{i=1}^{n}(Y_i-\mu_i)^2 = \displaystyle \sum_{i=1}^{n}(Y_i-\hat \mu_i)^2 + \displaystyle \sum_{i=1}^n (\hat \mu_i - \mu_i)^2$

It follows that:

$\displaystyle \sum_{i=1}^{n}(Y_i-\mu_i)^2\,\geq \sum_{i=1}^{n}(Y_i-\hat \mu_i)^2$

Considering only horizontal lines, and from equation (1):

$\large \displaystyle\sum_{i=1}^{n} (Y_i - \hat \beta_o)\,(\hat \beta_o - \beta_o)\,=\,(\hat \beta_o - \beta_o)\,\displaystyle\sum_{i=1}^{n}(Y_i - \hat \beta_o)=\,0$

This will happen if:

$\displaystyle\sum_{i=1}^{n}(Y_i - \hat \beta_o)=\,0$. In other words if $n\,\bar Y\,-\,n\,\hat \beta_o\,=\,0$, or

$\Large \bar Y = \beta_o \small \tag 2$.

If we do regression through the origin:

$\displaystyle\sum_{i=1}^{n} (Y_i - \hat \mu_i)\,(\hat \mu_i - \mu_i)\,=\,\displaystyle\sum_{i=1}^{n} (Y_i - \hat \beta_1\, X_i)\,(\hat \beta_1 X_i - \beta_1\,X_i)\,=\,\displaystyle\sum_{i=1}^{n}(Y_i\,\hat \beta_1\,X_i\,-\,Y_i\beta_1 X_i)(-\hat \beta_1 X_i \hat \beta_1 X_i\, +\,\hat\beta_1 X_i \beta_1 X_i)$

$=\,(\hat \beta_1 - \beta_1)\,\displaystyle\sum_{i=1}^{n}(Y_iX_i)-(\hat \beta_1 X_i^2)$. And this will be zero if:

$\displaystyle\sum_{i=1}^{n}(Y_iX_i)-(\hat \beta_1 X_i^2)=\displaystyle\sum_{i=1}^{n}(Y_iX_i)-\hat \beta_1\displaystyle\sum_{i=1}^{n}X_i^2=0$. Hence,

$\Large \hat \beta_1=\frac{\displaystyle\sum_{i=1}^{n}Y_iX_i}{\displaystyle\sum_{i=1}^{n}X_i^2} \small \tag 3$.

Now doing both intercept and slope:

$0 =\displaystyle\sum_{i=1}^{n} (Y_i - \hat \mu_i)\,(\hat \mu_i - \mu_i)\, =\displaystyle\sum_{i=1}^{n}(Y_i -\hat\beta_o - \hat \beta_1 X_i)(\hat\beta_o+\hat\beta_1X_i-\beta_o-\beta_1X_i)$

$= \displaystyle\sum_{i=1}^{n} (Y_i -\hat\beta_o \hat \beta_1 X_i)((\hat\beta_o-\beta_o)+\hat\beta_1X_i-\beta_1X_i)$

$=\large (\hat \beta_o-\beta_o)\displaystyle\sum_{i=1}^{n}(Y_i-\hat\beta_o-\hat\beta_1X_i)+(\hat\beta_1-\beta_1)\displaystyle\sum_{i=1}^{n}(Y_i-\hat\beta_o-\hat\beta_1X_i)X_i \small \tag 4$

$0 = \displaystyle\sum_{i=1}^{n}(Y_i-\hat\beta_o-\hat\beta_1X_i)=n\bar Y_i-n\hat\beta_o-n\hat\beta_1\bar X_i$. Hence,

$\Large \hat\beta_o\,=\,\bar Y\,-\,\hat\beta_1\,\bar X \small \tag 5$

And for the second part of equation (4):

$0=\displaystyle\sum_{i=1}^{n}(Y_i-\hat\beta_o-\hat\beta_1X_i)X_i$. Substituting $\hat\beta_o=\bar Y- \hat \beta_1 \bar X$ for $\hat\beta_o$:

$0=\displaystyle\sum_{i=1}^{n}(Y_i-\bar Y + \hat \beta_1 \bar X - \hat \beta_1X_i)X_i =\displaystyle\sum_{i=1}^{n}(Y_i-\bar Y)X_i - \hat\beta_1 \sum_{i=1}^{n}(X_i-\bar X)X_i$

Since $\displaystyle\sum_{i=1}^{n}(Y_i-\bar Y)=0$, also $\displaystyle\bar X\sum_{i=1}^{n}(Y_i-\bar Y)=0$.

$\large\hat \beta_1\,=\,\frac{\sum_{i=1}^{n}(Y_i-\bar Y)X_i}{\sum_{i=1}^{n}(X_i-\bar X)X_i} =\frac{\sum_{i=1}^{n}(Y_i-\bar Y)(X_i-\bar X)}{\sum_{i=1}^{n}(X_i-\bar X)(X_i-\bar X)}$

Checking equation (1):

$\large \frac{\sum_{i=1}^{n}(Y_i-\bar Y)(X_i-\bar X)}{\sum_{i=1}^{n}(X_i-\bar X)(X_i-\bar X)} =\frac{cov(X,Y)}{SD(X)}=\frac{\frac{cor(X,Y)}{SD(X)SD(Y)}}{SD(X)}$. Hence,

$\Large \hat\beta_1\,=\,cor(Y, X)\,\frac{SD(Y)}{SD(X)}\,=\,\frac{cov(Y,X)}{var(X)}\small \tag 6$