1
$\begingroup$

Suppose we use the least squares criterion to fit a linear model for the following dataset: $(x_1,y_1),...,(x_m,y_m)\in R \times R$, by solving the following optimisation problem:

$$(a^*,b^*) = \text{argmin}_{a,b}\sum^m_{i=1}(y_i-ax_i+b)^2$$

Assume that the solution is unique. Now, my question is, would any of the following statements be true?

i) $\sum^m_{i=1}(y_i-a^*x_i+b^*)y_i = 0$

ii) $\sum^m_{i=1}(y_i-a^*x_i+b^*)x_i^2 = 0$

iii) $\sum^m_{i=1}(y_i-a^*x_i+b^*)x_i = 0$

iv) $\sum^m_{i=1}(y_i-a^*x_i+b^*)^2 = 0$

By my understanding, we have to take the derivative of the loss function wrt $a$ and $b$:

$$\frac{\partial }{\partial a}\sum_{i=1}^m(y_i-ax_i+b)^2 = 0 \\ \sum_{i=1}^m\frac{\partial }{\partial a}(y_i-ax_i+b)^2 = 0 \\ 2(y_i-ax_i+b)\frac{\partial }{\partial a}(y_i-ax_i+b) = 0 \\ 2(y_i-ax_i+b)(-x_i)=0 \\ -2(y_i-ax_i+b)x_i = 0 \\ (y_i-ax_i+b)x_i=0$$

So this means that statement (iii) should be true.

And for $b$:

$$\frac{\partial }{\partial b}\sum_{i=1}^m(y_i-ax_i+b)^2 = 0 \\ \frac{\partial }{\partial b}(y_i-ax_i+b)^2 = 2(y_i-a_i+b)\frac{\partial }{\partial b} (y-ax_i+b) = 0 \\ 2(y-ax_i+b)(+1) = 0 \\ (y-ax_i+b) = 0$$

So this means that every statement is true.

Surely my second deduction is incorrect? What am I doing wrong here?

$\endgroup$
1
  • 1
    $\begingroup$ Your calculations are incorrect. Use the Chain Rule to find the derivatives. $\endgroup$
    – whuber
    Commented Aug 1, 2022 at 18:36

1 Answer 1

1
+50
$\begingroup$

$$ (y-ax_i+b) = 0$$

So this means that every statement is true.

Surely my second deduction is incorrect? What am I doing wrong here?

You forgot a summation sign. It should be

$$\sum_{i=1}^m (y-ax_i+b) = 0$$

The same happend in your first deduction which should end with

$$\sum_{i=1}^m (y-ax_i+b) x_i= 0$$

and not $ (y-ax_i+b)x_i = 0$

$\endgroup$
2
  • $\begingroup$ silly me, I forgot about that. So just to confirm, only (iii) would be correct, right? $\endgroup$
    – Slim Shady
    Commented Aug 5, 2022 at 11:14
  • $\begingroup$ Yes only iii is right. Sidenote: you can also write the two equations in matrix form $$X^T(Y - X\beta) = 0$$ where $X$ is a matrix with one colum with $x_i$ and one column with $1$'s and $\beta$ is a column vector with $a$ and $b$ as elements. That matrix form can be rewritten as $$X^T Y-X^TX\beta = 0$$ or $$X^TX\beta = X^T Y$$ and finally the more well known $$\beta = (X^TX)^{-1}X^T Y$$ $\endgroup$ Commented Aug 5, 2022 at 11:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.