# Deriving the normal equations' coefficients

Suppose we use the least squares criterion to fit a linear model for the following dataset: $$(x_1,y_1),...,(x_m,y_m)\in R \times R$$, by solving the following optimisation problem:

$$(a^*,b^*) = \text{argmin}_{a,b}\sum^m_{i=1}(y_i-ax_i+b)^2$$

Assume that the solution is unique. Now, my question is, would any of the following statements be true?

i) $$\sum^m_{i=1}(y_i-a^*x_i+b^*)y_i = 0$$

ii) $$\sum^m_{i=1}(y_i-a^*x_i+b^*)x_i^2 = 0$$

iii) $$\sum^m_{i=1}(y_i-a^*x_i+b^*)x_i = 0$$

iv) $$\sum^m_{i=1}(y_i-a^*x_i+b^*)^2 = 0$$

By my understanding, we have to take the derivative of the loss function wrt $$a$$ and $$b$$:

$$\frac{\partial }{\partial a}\sum_{i=1}^m(y_i-ax_i+b)^2 = 0 \\ \sum_{i=1}^m\frac{\partial }{\partial a}(y_i-ax_i+b)^2 = 0 \\ 2(y_i-ax_i+b)\frac{\partial }{\partial a}(y_i-ax_i+b) = 0 \\ 2(y_i-ax_i+b)(-x_i)=0 \\ -2(y_i-ax_i+b)x_i = 0 \\ (y_i-ax_i+b)x_i=0$$

So this means that statement (iii) should be true.

And for $$b$$:

$$\frac{\partial }{\partial b}\sum_{i=1}^m(y_i-ax_i+b)^2 = 0 \\ \frac{\partial }{\partial b}(y_i-ax_i+b)^2 = 2(y_i-a_i+b)\frac{\partial }{\partial b} (y-ax_i+b) = 0 \\ 2(y-ax_i+b)(+1) = 0 \\ (y-ax_i+b) = 0$$

So this means that every statement is true.

Surely my second deduction is incorrect? What am I doing wrong here?

• Your calculations are incorrect. Use the Chain Rule to find the derivatives.
– whuber
Aug 1, 2022 at 18:36

$$(y-ax_i+b) = 0$$

So this means that every statement is true.

Surely my second deduction is incorrect? What am I doing wrong here?

You forgot a summation sign. It should be

$$\sum_{i=1}^m (y-ax_i+b) = 0$$

The same happend in your first deduction which should end with

$$\sum_{i=1}^m (y-ax_i+b) x_i= 0$$

and not $$(y-ax_i+b)x_i = 0$$

• silly me, I forgot about that. So just to confirm, only (iii) would be correct, right? Aug 5, 2022 at 11:14
• Yes only iii is right. Sidenote: you can also write the two equations in matrix form $$X^T(Y - X\beta) = 0$$ where $X$ is a matrix with one colum with $x_i$ and one column with $1$'s and $\beta$ is a column vector with $a$ and $b$ as elements. That matrix form can be rewritten as $$X^T Y-X^TX\beta = 0$$ or $$X^TX\beta = X^T Y$$ and finally the more well known $$\beta = (X^TX)^{-1}X^T Y$$ Aug 5, 2022 at 11:21