RMSE vs MSE loss function - the optimization solutions are equivalent? If we optimize a function $f$ with respect to loss $L$, which is defined as RMSE; Are we going to get the same solution as optimizing MSE ? Even, if the function $f$ is non-linear (e.g. a neural network) ?
 A: RMSE is a square root of MSE. If you take the square root of a bunch of numbers, their relative ordering would not change. For optimization, what matters is the relative ordering of different solutions. Notice however that if you use penalties for regularization, e.g. $L_1$ or $L_2$, the solution may be different since the size of penalty relative to the raw loss would change.
A: In addition to @Tim's (+1) answer, I'd like to add some other perspective (also repeating some things). If some $x^*$ are the minimizers of RMSE ($\geq 0$), they're the minimizer of MSE, because the operation is monotonic, e.g. ($2 < 3 \rightarrow 2^2<3^2$). So, the same set of global optimizers, if there exists more than one, exist for the MSE.
But, optimization algorithms aren't perfect, and end up in local optima. A slight change in loss function will affect your gradient steps (e.g. the corresponding learning rates for the two loss functions will be different), therefore may lead you to another local optimum. So, it's quite possible to end up in different locations.
