If we optimize a function $f$ with respect to loss $L$, which is defined as RMSE; Are we going to get the same solution as optimizing MSE ? Even, if the function $f$ is non-linear (e.g. a neural network) ?
$\begingroup$
$\endgroup$
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
2
RMSE is a square root of MSE. If you take the square root of a bunch of numbers, their relative ordering would not change. For optimization, what matters is the relative ordering of different solutions. Notice however that if you use penalties for regularization, e.g. $L_1$ or $L_2$, the solution may be different since the size of penalty relative to the raw loss would change.
-
1$\begingroup$ One addition: it's true that the minimizers change in case if regularization, but a bit more is true. Even the minimizer of the regularized Problems are the same up to the choice of the regularization parameter! $\endgroup$– DirkJun 28, 2021 at 12:20
-
$\begingroup$ Thanks for answer, especially the notice about L1/L2 $\endgroup$ Jul 19, 2021 at 13:38
$\begingroup$
$\endgroup$
1
In addition to @Tim's (+1) answer, I'd like to add some other perspective (also repeating some things). If some $x^*$ are the minimizers of RMSE ($\geq 0$), they're the minimizer of MSE, because the operation is monotonic, e.g. ($2 < 3 \rightarrow 2^2<3^2$). So, the same set of global optimizers, if there exists more than one, exist for the MSE.
But, optimization algorithms aren't perfect, and end up in local optima. A slight change in loss function will affect your gradient steps (e.g. the corresponding learning rates for the two loss functions will be different), therefore may lead you to another local optimum. So, it's quite possible to end up in different locations.
-
2$\begingroup$ +1 good point in the "yes, but actually no" second paragraph. $\endgroup$– Tim ♦Jun 28, 2021 at 6:12