Working my way through a copy of Draper and Smith's Applied Regression Analysis. I am on the first chapter and looking at the section that involves deriving the OLS equations. No linear algebra has been used yet.

The section I am looking at has started by presenting the sum of squared deviations equation, which has then been differentiated for the intercept and slope parameters and set to 0 to end with the formula for the intercept:


And the slope:


The next step is to substitute $\beta_0$ for $b_0$ and $\beta_1$ for $b_1$ which apparently leads you to:




I really am at a loss in understanding how the authors got from the first two equations to the second two. They say that they simply substituted $β_0$ for $b_0$ and $β_1$ for $b_1$. Not sure how this changes anything. Can anyone explain?

  • 1
    $\begingroup$ Where you say "next step is to substitute" ... you don't then do any such substitution. If you did there would no longer be $b$'s there. It seems like you must have a mistake in what you're writing (either the substitution isn't being done there or you copied the rest wrongly). However, the last 2 equations follow from the previous ones simply by splitting up the sum; this is completely straightforward algebraic manipulation of the kind $(a-b)x = ax-bx$. Can you explain why the sums go from 0 in the lower limit? (i.e. how is there a zero-th observation?) $\endgroup$ – Glen_b Jan 17 '17 at 4:40
  • $\begingroup$ $b_0$ and $b_1$ are just notation for the least squares estimates of the parameters $beta_0$ and $beta_1$ respectively. $\endgroup$ – Michael R. Chernick Jan 17 '17 at 4:42
  • $\begingroup$ I know. So how do they derive the second set of equations from the first. Thanks. $\endgroup$ – user183974 Jan 17 '17 at 4:43
  • $\begingroup$ Thanks @Glen_b. I almost understand. But I am not sure where $nb_0$ comes from in the second equation. Can you please explain? Is the zeroth observation there because it is the intercept? $\endgroup$ – user183974 Jan 17 '17 at 4:59
  • 1
    $\begingroup$ $\sum b_0 = b_0 + b_0 + b_0 + ... + b_0 = n b_0$... but only if the lower limit of the sum is 1 not 0. Please check the original mathematics carefully. $\endgroup$ – Glen_b Jan 17 '17 at 5:03

It's nothing but basic algebra.

For the first equation:

\begin{align*} \sum_{i=1}^n(Y_i-b_0-b_1X_i) &= \sum_{i=1}^nY_i-\sum_{i=1}^nb_0-\sum_{i=1}^nb_1X_i \\ &= \sum_{i=1}^nY_i-nb_0-b_1\sum_{i=1}^nX_i \end{align*}

For the second equation:

\begin{align*} \sum_{i=1}^nX_i(Y_i-b_0-b_1X_i) &= \sum_{i=1}^n(X_iY_i-b_0X_i-b_1X^2_i) \\ &= \sum_{i=1}^nX_iY_i-\sum_{i=1}^n b_0X_i-\sum_{i=1}^n b_1X^2_i \\ &= \sum_{i=1}^nX_iY_i-b_0\sum_{i=1}^n X_i-b_1\sum_{i=1}^n X^2_i \end{align*}

$b_0$ and $b_1$ don't depend on $i$ can be pulled out of the summation via the distributive property.

For example:

\begin{align*} \sum_{i=1}^n b x_i &= b x_1 + b x_2 + b x_3 + \ldots + b x_n \\ &= b\left( x_1 + x_2 + x_3 + \ldots + x_n \right)\\ &= b \sum_{i=1}^n x_i \end{align*}

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.