Ridge regression to minimize RMSE instead of MSE

Question

Cross-posted from my identical question on math.stackexchange:

Given a metrix $X$ and a vector $\vec{y}$, ordinary least squares (OLS) regression tries to find $\vec{c}$ such that $\left\| X \vec{c} - \vec{y} \right\|_2^2$ is minimal. (If we assume that $\left\| \vec{v}\right\|_2^2=\vec{v} \cdot \vec{v}$.)

Ridge regression tries to find $\vec{c}$ such that $\left\| X \vec{c} - \vec{y} \right\|_2^2 + \left\| \Gamma \vec{c} \right\|_2^2 $ is minimal.

However, I have an application where I need to minimize not the sum of squared errors, but the square root of this sum. Naturally, the square root is an increasing function, so this minimum will be at the same location, so the OLS regression will still give the same result. But will ridge regression?

On the one hand, I don't see how minimizing $\left\| X \vec{c} - \vec{y} \right\|_2^2 + \left\| \Gamma \vec{c} \right\|_2^2 $ will necessarily result in the same $\vec{c}$ as minimizing $\sqrt{ \left\| X \vec{c} - \vec{y} \right\|_2^2 } + \left\| \Gamma \vec{c} \right\|_2^2 $.

On the other hand, I've read (though never seen shown) that minimizing $\left\| X \vec{c} - \vec{y} \right\|_2^2 + \left\| \Gamma \vec{c} \right\|_2^2 $ (ridge regression) is identical to minimizing $\left\| X \vec{c} - \vec{y} \right\|_2^2$ under the constraint that $ \left\|\Gamma \vec{c}\right\|_2^2 < t$, where $t$ is some parameter. And if this is the case, then it should result in the same solution as minimizing $\sqrt{ \left\| X \vec{c} - \vec{y} \right\|_2^2}$ under the same constraint.

Answer 1

minimizing

$$ \left\| X \vec{c} - \vec{y} \right\|_2^2 + \left\| \Gamma \vec{c} \right\|_2^2 $$

and minimizing

$$ \sqrt{\left\| X \vec{c} - \vec{y} \right\|_2^2} + \left\| \Gamma \vec{c} \right\|_2^2 $$

do not directly relate to minimizing ${\left\| X \vec{c} - \vec{y} \right\|_2^2}$ or $\sqrt{\left\| X \vec{c} - \vec{y} \right\|_2^2}$ under the constraint $\left\|\vec{c}\right\|_2^2 < t$.

There will need to be a conversion between $t$ and $\Gamma$ which will be different for the two different cost functions. Thus the minimization of MSE and RMSE with a same penalty term defined by $\Gamma$ will relate to a constrained minimization with different constraints $t$.

^{Note that for every solution $\vec{c}$ to minimizing the MSE with penalty term $\Gamma_1$ there will be a penalty term $\Gamma_2$ that results in the same solution $\vec{c}$ when minimizing the penalized RMSE. So for many practical purposes you can use any methods/software that solves the penalized MSE problem, but only you need to use a different cost function when, for instance, performing cross validation to select the ideal $\Gamma$.}