This example was taken from Mathematical Statistics : A Unified Introduction (ISBN 9780387227696), page 58, under the section 'The Principle of Least Squares'. I think my problem has more to do with algebra but since this is a statistics book, I thought I should post here...

Anyway, I need to estimate k base on this equation:

alt text

The goal is to reach here:

alt text

I managed to follow the example until I got stuck here: alt text

The explanation to reach to the goal from above is described as follows: alt text

I know how to get k but I totally don't understand how the 'middle term' was eliminated. Please help. I will provide more details if needed.


There is a constraint imposed a few lines above the highlighted text which states:

$\sum_i{v_i (d_i-k v_i)} = 0$

Does the that help?


In response to your second comment:

Consider: $\sum_i{(d_i - l v_i)^2}$. This can be re-written as:

$\sum_i{((d_i - k v_i) + (k v_i - l v_i))^2}$

Expanding the square, we have:

$\sum_i{(d_i - k v_i)^2 + \sum_i(k v_i - l v_i)^2 + \sum_i{2 (d_i - k v_i) (k v_i - l v_i)}}$

Simplifying the above, we have:

$\sum_i{(d_i - k v_i)^2 + (k-l)^2 \sum_i{v_i^2} + 2 (k-l) \sum_i{v_i (d_i - k v_i)}}$

So, if we choose $k$ such that $\sum_i{v_i (d_i-k v_i)} = 0$ then it follows that:

$\sum_i{(d_i - l v_i)^2} = \sum_i{(d_i - k v_i)^2} + (k-l)^2 \sum_i{v_i^2}$

But, then it follows that:

$\sum_i{(d_i - l v_i)^2} > \sum_i{(d_i - k v_i)^2}$

as long as $k \ne l$.

Thus, what we have shown is the following: If $k$ satisfies the constraint we imposed then it must be the case that the corresponding SSE is less than the SSE for any other $l$ that we can choose. Thus, the value of $k$ that satisfies the constraint is our least squares estimate.

  • $\begingroup$ I'm sorry, it does not. I don't know how 2(k-1)sum(vi(di-kvi)) can become sum(di-kvi)^2 after k was estimated. Obviously, I have failed to grasp the significance of the constraint imposed. Please explain the significance of the constraint's role in solving the equation...Thank you. $\endgroup$ – Sara Oct 21 '10 at 13:21
  • $\begingroup$ "The idea is the following insight: Suppose that we choose k such that the above constraint is satisfied then it immediately follows that: ∑i(di−lvi)2=∑i(d−kvi)2+(k−l)2∑iv2i " Could you please show the intermediate steps? I really cannot see how that equation follows immediately from satisfying the constraint. Sorry for the inconvenience caused. $\endgroup$ – Sara Oct 22 '10 at 0:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.