I seem to be misunderstanding a claim about linear regression methods that I've seen in various places. The parameters of the problem are:
Input:
$N$ data samples of $p+1$ quantities each consisting of a "response" quantity $y_i$ and $p$ "predictor" quantities $x_{ij}$
The result desired is a "good linear fit" which predicts the response based on the predictors where a good fit has small differences between the prediction and the observed response (among other criteria).
Output: $p+1$ coefficients $\beta_j$ where $\beta_0 + \sum_{j=1}^p x_{ij} * \beta_j$ is a "good fit" for predicting the response quantity from the predictor quantities.
I'm confused about the "ridge regression" approach to this problem. In "The Elements of Statistical Learning" by Hastie, Tibshirani, and Friedman page 63 ridge regression is formulated in two ways.
First as the constrained optimization problem:
$$ \text{argmin}_\beta \sum_{i=1}^N { ( y_i - (\beta_0 + \sum_{j=1}^p (x_{ij} * \beta_j)) )^2 } $$ subject to the constraint $$ \sum_{j=1}^p \beta_i^2 \leq t $$ for some positive parameter t.
Second is the penalized optimization problem: $$ \text{argmin}_\beta ( \lambda \sum_{j=1}^p { \beta_j^2 } ) + \sum_{i=1}^N { ( y_i - (\beta_0 + \sum_{j=1}^p (x_{ij} * \beta_j)) )^2 } $$ for some positive parameter $\lambda$.
The text says that these formulations are equivalent and that there is a "one to one correspondence between the parameters $\lambda$ and $t$". I've seen this claim (and similar ones) in several places in addition to this book. I think I am missing something because I don't see how the formulations are equivalent as I understand it.
Consider the case where $N=2$ and $p=1$ with $y_1=0$, $x_{1,1}=0$ and $y_2=1$, $x_{1,2}=1$. Choosing the parameter $t=2$ the constrained formulation becomes:
$$ \text{argmin}_{\beta_0,\beta_1} ( \beta_0^2 + (1 - (\beta_0 + \beta_1))^2 ) $$
expanded to
$$ \text{argmin}_{\beta_0,\beta_1} ( 2 \beta_{0}^{2} + 2 \beta_{0} \beta_{1} - 2 \beta_{0} + \beta_{1}^{2} - 2 \beta_{1} + 1 ) $$
To solve this find the solution where the partial derivatives with respect to $\beta_0$ and $\beta_1$ are zero: $$ 4 \beta_{0} + 2 \beta_{1} - 2 = 0 $$ $$ 2 \beta_{0} + 2 \beta_{1} - 2 = 0 $$ with solution $\beta_0 = 0$ and $\beta_1 = 1$. Note that $\beta_0^2 + \beta_1^2 \le t$ as required.
How does this derivation relate to the other formulation? According to the explanation there is some value of $\lambda$ uniquely corresponding to $t$ where if we optimize the penalized formulation of the problem we will derive the same $\beta_0$ and $\beta_1$. In this case the penalized form becomes $$ \text{argmin}_{\beta_0,\beta_1} ( \lambda (\beta_0^2 + \beta_1^2) + \beta_0^2 + (1 - (\beta_0 + \beta_1))^2 ) $$ expanded to $$ \text{argmin}_{\beta_0,\beta_1} ( \beta_{0}^{2} \lambda + 2 \beta_{0}^{2} + 2 \beta_{0} \beta_{1} - 2 \beta_{0} + \beta_{1}^{2} \lambda + \beta_{1}^{2} - 2 \beta_{1} + 1 ) $$ To solve this find the solution where the partial derivatives with respect to $\beta_0$ and $\beta_1$ are zero: $$ 2 \beta_{0} \lambda + 4 \beta_{0} + 2 \beta_{1} - 2 = 0 $$ $$ 2 \beta_{0} + 2 \beta_{1} \lambda + 2 \beta_{1} - 2 = 0 $$ for these equations I get the solution $$ \beta_0 = \lambda/(\lambda^2 + 3\lambda + 1) $$ $$ \beta_1 = (\lambda + 1)/((\lambda + 1)(\lambda + 2) - 1) $$ If that is correct the only way to get $\beta_0 = 0$ is to set $\lambda = 0$. However that would be the same $\lambda$ we would need for $t = 4$, so what do they mean by "one to one correspondence"?
In summary I'm totally confused by the two presentations and I don't understand how they correspond to each other. I don't understand how you can optimize one form and get the same solution for the other form or how $\lambda$ is related to $t$. This is just one instance of this kind of correspondence -- there are others for other approaches such as lasso -- and I don't understand any of them.
Someone please help me.