Deriving the normal equations' coefficients

Question

Suppose we use the least squares criterion to fit a linear model for the following dataset: $(x_1,y_1),...,(x_m,y_m)\in R \times R$, by solving the following optimisation problem:

$$(a^*,b^*) = \text{argmin}_{a,b}\sum^m_{i=1}(y_i-ax_i+b)^2$$

Assume that the solution is unique. Now, my question is, would any of the following statements be true?

i) $\sum^m_{i=1}(y_i-a^*x_i+b^*)y_i = 0$

ii) $\sum^m_{i=1}(y_i-a^*x_i+b^*)x_i^2 = 0$

iii) $\sum^m_{i=1}(y_i-a^*x_i+b^*)x_i = 0$

iv) $\sum^m_{i=1}(y_i-a^*x_i+b^*)^2 = 0$

By my understanding, we have to take the derivative of the loss function wrt $a$ and $b$:

$$\frac{\partial }{\partial a}\sum_{i=1}^m(y_i-ax_i+b)^2 = 0 \\ \sum_{i=1}^m\frac{\partial }{\partial a}(y_i-ax_i+b)^2 = 0 \\ 2(y_i-ax_i+b)\frac{\partial }{\partial a}(y_i-ax_i+b) = 0 \\ 2(y_i-ax_i+b)(-x_i)=0 \\ -2(y_i-ax_i+b)x_i = 0 \\ (y_i-ax_i+b)x_i=0$$

So this means that statement (iii) should be true.

And for $b$:

$$\frac{\partial }{\partial b}\sum_{i=1}^m(y_i-ax_i+b)^2 = 0 \\ \frac{\partial }{\partial b}(y_i-ax_i+b)^2 = 2(y_i-a_i+b)\frac{\partial }{\partial b} (y-ax_i+b) = 0 \\ 2(y-ax_i+b)(+1) = 0 \\ (y-ax_i+b) = 0$$

So this means that every statement is true.

Surely my second deduction is incorrect? What am I doing wrong here?

Answer 1

$$ (y-ax_i+b) = 0$$

So this means that every statement is true.

Surely my second deduction is incorrect? What am I doing wrong here?

You forgot a summation sign. It should be

$$\sum_{i=1}^m (y-ax_i+b) = 0$$

The same happend in your first deduction which should end with

$$\sum_{i=1}^m (y-ax_i+b) x_i= 0$$

and not $ (y-ax_i+b)x_i = 0$