9
$\begingroup$

If we optimize a function $f$ with respect to loss $L$, which is defined as RMSE; Are we going to get the same solution as optimizing MSE ? Even, if the function $f$ is non-linear (e.g. a neural network) ?

$\endgroup$

2 Answers 2

23
$\begingroup$

RMSE is a square root of MSE. If you take the square root of a bunch of numbers, their relative ordering would not change. For optimization, what matters is the relative ordering of different solutions. Notice however that if you use penalties for regularization, e.g. $L_1$ or $L_2$, the solution may be different since the size of penalty relative to the raw loss would change.

$\endgroup$
2
  • 1
    $\begingroup$ One addition: it's true that the minimizers change in case if regularization, but a bit more is true. Even the minimizer of the regularized Problems are the same up to the choice of the regularization parameter! $\endgroup$
    – Dirk
    Jun 28, 2021 at 12:20
  • $\begingroup$ Thanks for answer, especially the notice about L1/L2 $\endgroup$ Jul 19, 2021 at 13:38
16
$\begingroup$

In addition to @Tim's (+1) answer, I'd like to add some other perspective (also repeating some things). If some $x^*$ are the minimizers of RMSE ($\geq 0$), they're the minimizer of MSE, because the operation is monotonic, e.g. ($2 < 3 \rightarrow 2^2<3^2$). So, the same set of global optimizers, if there exists more than one, exist for the MSE.

But, optimization algorithms aren't perfect, and end up in local optima. A slight change in loss function will affect your gradient steps (e.g. the corresponding learning rates for the two loss functions will be different), therefore may lead you to another local optimum. So, it's quite possible to end up in different locations.

$\endgroup$
1
  • 2
    $\begingroup$ +1 good point in the "yes, but actually no" second paragraph. $\endgroup$
    – Tim
    Jun 28, 2021 at 6:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.