If we optimize a function $f$ with respect to loss $L$, which is defined as RMSE; Are we going to get the same solution as optimizing MSE ? Even, if the function $f$ is non-linear (e.g. a neural network) ?
$\begingroup$
$\endgroup$
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
5
RMSE is a square root of MSE. If you take the square root of a bunch of numbers, their relative ordering would not change. For optimization, what matters is the relative ordering of different solutions. Notice however that if you use penalties for regularization, e.g. $L_1$ or $L_2$, the solution may be different since the size of penalty relative to the raw loss would change.
-
1$\begingroup$ One addition: it's true that the minimizers change in case if regularization, but a bit more is true. Even the minimizer of the regularized Problems are the same up to the choice of the regularization parameter! $\endgroup$– DirkCommented Jun 28, 2021 at 12:20
-
$\begingroup$ Thanks for answer, especially the notice about L1/L2 $\endgroup$ Commented Jul 19, 2021 at 13:38
-
1$\begingroup$ Does it affect the gradient, though? Is gradient descent with MSE the same as gradient descent with RMSE? $\endgroup$– JacKeownCommented Jul 7, 2022 at 0:50
-
$\begingroup$ @JacKeown yes, gradient descent is an optimization problem that can be used with any differential function. $\endgroup$– TimCommented Jul 7, 2022 at 5:53
-
$\begingroup$ @Tim, I meant, "Is the RMSE gradient proportional to the MSE gradient?", not "Can you apply gradient descent to both?". $\endgroup$– JacKeownCommented Jul 8, 2022 at 15:55
$\begingroup$
$\endgroup$
1
In addition to @Tim's (+1) answer, I'd like to add some other perspective (also repeating some things). If some $x^*$ are the minimizers of RMSE ($\geq 0$), they're the minimizer of MSE, because the operation is monotonic, e.g. ($2 < 3 \rightarrow 2^2<3^2$). So, the same set of global optimizers, if there exists more than one, exist for the MSE.
But, optimization algorithms aren't perfect, and end up in local optima. A slight change in loss function will affect your gradient steps (e.g. the corresponding learning rates for the two loss functions will be different), therefore may lead you to another local optimum. So, it's quite possible to end up in different locations.
-
3$\begingroup$ +1 good point in the "yes, but actually no" second paragraph. $\endgroup$– TimCommented Jun 28, 2021 at 6:12