Geometric interpretation of penalized linear regression II

Question

An older question gives an intuitive explanation of how penalized linear regression works, using two separate contours: one for the least square objective, one for the penalty term (i.e. regularization).

It also says that the sum of the least squares and regularization objectives is minimized when the contours meet for the first time. It seems, however, that to "calculate" the point where they meet "for the first time", we need to know the rate of growth of both contours. Unfortunately, I can't figure out how to arrive at it.

UPDATE: At the bottom of page 5 of these notes on variable selection and regularization, the authors say that $$ argmin \sum{(y_i - \alpha - \beta'x_i)^2}$$ subject to $$\sum{|\beta_j|} < t$$ is equivalent to minimizing $$\frac{1}{2n}\sum{(y_i - \alpha - \beta'x_i)^2} + \lambda \sum{|\beta_j|}$$.

If we look at the contour growing from this point of view, it means that:

1. We fix the contour plot for the penalty term, somehow (?) calculating $t$ from $\lambda$.
2. We only grow the contour for the least square objective.

Is this reasoning correct? If yes, could you point me to some literature that discusses this in more detail, especially:

1. The equivalence between the minimization of the function with inequality constraints and the minimization of the objective function in the more familiar form with the $\lambda$ parameter.
2. The calculation of $t$ given $\lambda$.

Answer 1

The equivalence between penalized form (with lambda) and constrained form (with t) follows from Lagrange duality, please look up any optimization textbook. (NOTE: constraint should be non-strict inequality!)

The bad news is, there is no straightforward calculation of t given lambda or vice verse. You need to solve one optimization problem, only then can the parameter of the other be found. If penalized form has been solved and yielded $\beta^*$, then it's easy to see that $t=\sum |\beta^*_j|$. Vice verse isn't that easy, but it will fall out of almost any optimization algorithm (otherwise need to work through Karush-Kuhn-Tucker conditions).