Showing what the best line of fit is given the method of least squares - what do I do from here?

Question

I have been working through Multivariable Calculus 4th Edition by James Stewart (©1999) and am currently stuck on what seems to be a stats problem on problem 51 of Chapter 15.7:

51. Suppose that a scientist has reason to believe that two quantities $x$ and $y$ are related linearly, that is, $y=mx+b$, at least approximately, for some values of $m$ and $b$. The scientist performs an experiment and collects data in the form of point $(x_1,y_1),(x_2,y_2),...,(x_n,y_n)$, and then plots these points. The points don't exactly lie on a straight line, so the scientist wants to find constants $m$ and $b$ so that the line $y=mx+b$ "fits" the points as well as possible. Let $d_i=y_i-(mx_i+b)$ be the vertical derivation from the line [$mx+b]$. The method of least squares determines $m$ and $b$ so as to minimize $\sum_{i=1}^{n}d_i^2$, the sum of the squares of these derivations. Show that, according to this method, the line of best fit is obtained when$$m\sum_{i=1}^nx_i+bn=\sum_{i=1}^ny_i\\m\sum_{i=1}^nx_i^2+b\sum_{i=1}^nx_i=\sum_{i=1}^nx_iy_i$$Thus, the line is found by solving these two equations in the two unknowns $m$ and $b$.

My thinking:

---

Mainly going off of how this chapter has been focusing on partial derivatives, what I could do is start by setting the gradient to 0 (defining our equation for line of best fit to be $\beta_1+\beta_2x$):$$\dfrac{\partial S}{\partial\beta_j}=2\sum_id_i\dfrac{\partial d_i}{\partial\beta_j}=0,j=1,2$$but now since $d_i=y_i-(\beta_1+\beta_2x)=y_i-f(x_i,\beta)$, we have that$$-2\sum_id_i\dfrac{\partial f(x_i,\beta)}{\partial b_j}=0,j=1,2$$Now, I believe that I am likely in a position where if I could arrange our two conditions into something similar to the equation that I have gotten from setting the gradient to 0, then I can solve this.

However, it seems that no matter what I try from here, I end up writing myself into a dead end, so my question is

How do I go about showing that the line of best fit using the method of least squares is obtained when$$m\sum_{i=1}^nx_n+bn=\sum_{i=1}^ny_i\\m\sum_{i=1}^nx_i^2+b\sum_{i=1}^nx_i=\sum_{i=1}^nx_iy_i$$

Answer 1

You need to find $m$ and $b$ that minimizes the sum of the squares errors, that is, the function: \begin{equation} S(m, b) = \sum_{i=1}^{n}d_i^2 = \sum_{i=1}^{n}(y_i-mx_i-b)^2. \end{equation} Note that \begin{equation} \frac{\partial S(m, b)}{\partial m} = 2\sum_{i=1}^{n}(y_i-mx_i-b)(-x_i) = 0 \ \implies \ \sum_{i=1}^{n}x_iy_i=m\sum_{i=1}^{n}x_i^2+b\sum_{i=1}^{n}x_i \end{equation} and \begin{equation} \frac{\partial S(m, b)}{\partial b} = 2\sum_{i=1}^{n}(y_i-mx_i-b)(-1) = 0 \ \implies \ \sum_{i=1}^{n}y_i=m\sum_{i=1}^{n}x_i+nb. \end{equation} These are the conditions from the statement. I imagine you know how to use the chain rule to obtain the derivatives above. Additionally, remember to apply the summation to each term individually.

Answer 2

but now since $d_i=y_i-(\beta_1+\beta_2x)=y_i-f(x_i,\beta)$, we have that$$-2\sum_id_i\dfrac{\partial f(x_i,\beta)}{\partial b_j}=0,j=1,2$$

That switch to a more general formula with $f(x_i,\beta)$ made it more confusing or complicated than necessary.

The expressions $\dfrac{\partial f(x_i,\beta)}{\partial \beta_j}$ are not so difficult when you take a step back and substitute $f(x_i,\beta) = \beta_1+\beta_2x_i$.

You have

$$\dfrac{\partial f(x_i,\beta)}{\partial \beta_1} = \dfrac{\partial (\beta_1+\beta_2x_i) }{\partial \beta_1} = 1$$

and

$$\dfrac{\partial f(x_i,\beta)}{\partial \beta_2} = \dfrac{\partial (\beta_1+\beta_2x_i) }{\partial \beta_2} = x_i$$

So you can get to the two equations

$$-2\sum_i d_i = -2\sum_i (\beta_1 + \beta_2 x_i) = 0$$ and $$-2\sum_i d_i x_i = -2\sum_i (\beta_1 + \beta_2 x_i) x_i = 0$$