How exactly to compute the ridge regression penalty parameter given the constraint?

Question

The accepted answer in this thread does a great job of showing that there is a one-to-one correspondence between $c$ and $\lambda$ in the two formulations of the ridge regression: $$ \underset{\beta}{min}(y-X\beta)^T(y-X\beta) + \lambda\beta^T\beta $$ and $$ \underset{\beta}{min}(y-X\beta)^T(y-X\beta) \text{ s.t. }\beta^T\beta\leq{c} $$

The linked answer shows this in the orthogonal case.

In a general (non-orthogonal case), how can I compute $\lambda$ from $c$?

Update Here is an answer for going from $\lambda$ to $c$:

Assuming that the coefs are constrained by the penalty, $$ \beta^T\beta = c $$ and $$ \beta = (X^TX + \lambda I)^{-1}X^Ty\\ \beta^T\beta = c = \beta^T(X^TX + \lambda I)^{-1}X^Ty $$

Still working on going the other way

Answer 1

Form the Lagrangian

$$ (\mathbf{y} - X \mathbf{B})'(\mathbf{y} - X \mathbf{B}) + \lambda (\mathbf{b}'\mathbf{b} - c)$$

Your $\lambda$ for the first problem is essentially the same $\lambda$ that's a Lagrange multiplier in the second problem.

If you solve the second problem, using some convex optimization software, you should be able to get the Lagrange multiplier (aka dual variable) associated with the constraint $\mathbf{b}'\mathbf{b} \leq c$.

Eg. for the CVX software package, search down this page until you get to the section on dual variables.

Towards a closed form solution...

Combining the first order conditions (see comments), you get:

\begin{align*} c &= \mathbf{y}'X\left(X'X + \lambda I \right)^{-2}X'\mathbf{y} \\ &= \mathbf{y}'X\left(Q \Lambda Q^{-1} + \lambda QQ^{-1} \right)^{-2}X'\mathbf{y} \quad \quad \text{by applying eigenvalue decomposition} \\ &= \mathbf{y}'X\left[Q \left( \Lambda + \lambda I \right) Q^{-1} \right]^{-2}X'\mathbf{y}\\ &= \mathbf{y}'X Q \left( \Lambda + \lambda I \right)^{-2} Q^{-1}X'\mathbf{y} \end{align*}

Let $\mathbf{w} = Q^{-1} X'\mathbf{y}$. Then $$ c = \mathbf{w'} \left( \Lambda + \lambda I \right) ^{-2} \mathbf{w} $$

Where $Q\Lambda Q^{-1} = X'X$ is the eigenvalue decomposition of $X'X$ and $\Lambda_i$ is the $i$th element of diagonal matrix $\Lambda$. (I think this is right, but I haven't checked my work.)

$$ c = \sum_i \frac{w_i^2}{\left( \Lambda_i + \lambda \right) ^2} $$

Then solving for $\lambda$ from $c$ looks like a polynomial mess?