# Deriving Least Squares formula

Working my way through a copy of Draper and Smith's Applied Regression Analysis. I am on the first chapter and looking at the section that involves deriving the OLS equations. No linear algebra has been used yet.

The section I am looking at has started by presenting the sum of squared deviations equation, which has then been differentiated for the intercept and slope parameters and set to 0 to end with the formula for the intercept:

$$\sum_{i=0}^n(Y_i-b_0-b_1X_i)=0$$

And the slope:

$$\sum_{i=0}^nX_i(Y_i-b_0-b_1X_i)=0$$

The next step is to substitute $\beta_0$ for $b_0$ and $\beta_1$ for $b_1$ which apparently leads you to:

$$\sum_{i=0}^nY_i-nb_0-b_1\sum_{i=0}^nX_i=0$$

And

$$\sum_{i=0}^nX_iY_i-b_0\sum_{i=0}^nX_i-b_1\sum_{i=0}^nX_i^2=0$$

I really am at a loss in understanding how the authors got from the first two equations to the second two. They say that they simply substituted $β_0$ for $b_0$ and $β_1$ for $b_1$. Not sure how this changes anything. Can anyone explain?

• Where you say "next step is to substitute" ... you don't then do any such substitution. If you did there would no longer be $b$'s there. It seems like you must have a mistake in what you're writing (either the substitution isn't being done there or you copied the rest wrongly). However, the last 2 equations follow from the previous ones simply by splitting up the sum; this is completely straightforward algebraic manipulation of the kind $(a-b)x = ax-bx$. Can you explain why the sums go from 0 in the lower limit? (i.e. how is there a zero-th observation?) – Glen_b -Reinstate Monica Jan 17 '17 at 4:40
• $b_0$ and $b_1$ are just notation for the least squares estimates of the parameters $beta_0$ and $beta_1$ respectively. – Michael R. Chernick Jan 17 '17 at 4:42
• I know. So how do they derive the second set of equations from the first. Thanks. – user183974 Jan 17 '17 at 4:43
• Thanks @Glen_b. I almost understand. But I am not sure where $nb_0$ comes from in the second equation. Can you please explain? Is the zeroth observation there because it is the intercept? – user183974 Jan 17 '17 at 4:59
• $\sum b_0 = b_0 + b_0 + b_0 + ... + b_0 = n b_0$... but only if the lower limit of the sum is 1 not 0. Please check the original mathematics carefully. – Glen_b -Reinstate Monica Jan 17 '17 at 5:03

It's nothing but basic algebra.

For the first equation:

\begin{align*} \sum_{i=1}^n(Y_i-b_0-b_1X_i) &= \sum_{i=1}^nY_i-\sum_{i=1}^nb_0-\sum_{i=1}^nb_1X_i \\ &= \sum_{i=1}^nY_i-nb_0-b_1\sum_{i=1}^nX_i \end{align*}

For the second equation:

\begin{align*} \sum_{i=1}^nX_i(Y_i-b_0-b_1X_i) &= \sum_{i=1}^n(X_iY_i-b_0X_i-b_1X^2_i) \\ &= \sum_{i=1}^nX_iY_i-\sum_{i=1}^n b_0X_i-\sum_{i=1}^n b_1X^2_i \\ &= \sum_{i=1}^nX_iY_i-b_0\sum_{i=1}^n X_i-b_1\sum_{i=1}^n X^2_i \end{align*}

$b_0$ and $b_1$ don't depend on $i$ can be pulled out of the summation via the distributive property.

For example:

\begin{align*} \sum_{i=1}^n b x_i &= b x_1 + b x_2 + b x_3 + \ldots + b x_n \\ &= b\left( x_1 + x_2 + x_3 + \ldots + x_n \right)\\ &= b \sum_{i=1}^n x_i \end{align*}