$$\begin{align}
f(\beta) &= \lVert y-X\beta \lVert^2 \\
g(\beta) &= \lVert \beta \lVert^2\\
h(\beta) &= \lVert X\beta \lVert^2
\end{align}$$
normal
which
which can be viewed, geometrically, as finding the smallest ellipsoid $f(\beta)=RSS$$f(\beta)=\text{RSS }$ that touches the intersection of the sphere $g(\beta) = t$ and the ellipsoid $h(\beta)=1$
##Further notes regarding the limit $\lambda \to \infty$
- The usual ridge regression limit for $\lambda$ to infinity corresponds to a different point in the constrained ridge regression.
This 'old' limit corresponds to the point where $\mu$ is equal to -1.
Then the derivative of the Lagrange function in the normalized problem
The usual ridge regression limit for $\lambda$ to infinity corresponds to a different point in the constrained ridge regression.
This 'old' limit corresponds to the point where $\mu$ is equal to -1.
Then the derivative of the Lagrange function in the normalized problem
$$2 (1+\mu) X^{T}X \beta + 2 X^T y + 2 \lambda \beta$$ corresponds
to a solution for the derivative of the Lagrange function in the
standard problem
$$2 X^{T}X \beta^\prime + 2 X^T y + 2 \frac{\lambda}{(1+\mu)}
\beta^\prime \qquad \text{with $\beta^\prime = (1+\mu)\beta$}$$
- You could parameterize the 2-d problem as following:
$$\beta(\theta) = \begin{pmatrix} a \cos(\theta) \\ b
\sin(\theta) \end{pmatrix} $$
e.g. the regression is with:
$$ X = \begin{pmatrix} 1/a & 0 \\ 0 & 1/b \\ \end{pmatrix}$$
Then the following is to be minimized $$f(\theta) =
\left(\cos(\theta)-\frac{\hat{\beta}_{LS}}{a}\right)^2 +
\left(\sin(\theta)-\frac{\hat{\beta}_{LS}}{b}\right)^2 + \lambda
\left( a^2 \cos(\theta)^2 + b^2 \sin(\theta)^2 \right) $$ It is a
bit more work to simplify those trigonometric functions, but note that
the part $$\left( a^2 \cos(\theta)^2 + b^2 \sin(\theta)^2 \right)$$
has slope zero in the points $\theta = \frac{k}{2}\pi$, but the part
$$\left(\cos(\theta)-\frac{\hat{\beta}_{LS}}{a}\right)^2 +
\left(\sin(\theta)-\frac{\hat{\beta}_{LS}}{b}\right)^2 $$ does not. So
no matter what the value of $\lambda$ the solution does not have a
minimum in $\theta = \frac{k}{2}\pi$, although we can get arbitrarily
close when $\lambda \to \infty$.
