1
$\begingroup$

is there a characterization or an upper bound on the norm of the ridge regression estimator (coefficients)? As the Tikhonov regularization attempts to regularize these coefficients as part of the loss function, is there an exact form for this norm after the estimator is computed?

$\endgroup$
1
  • 1
    $\begingroup$ There is a closed form solution to the ridge regression problem, hence a closed form expression for the norm. $\endgroup$
    – passerby51
    Jan 8 at 6:52
1
$\begingroup$

We have $$ \hat\beta = (X^TX + \lambda I)^{-1}X^Ty $$ with $X \in \mathbb R^{n\times p}$ so, as @passerby51 commented, $$ \|\hat\beta\|^2 = y^TX(X^TX + \lambda I)^{-2}X^Ty. $$ To interpret this, let $X = UDV^T$ be the SVD of $X$. Then $$ \|\hat\beta\|^2 = y^T UD(D^2 + \lambda I)^{-2}DU^Ty = \sum_{i=1}^p \frac{d_i^2}{(d_i^2 + \lambda)^2} z_i^2 $$ with $z = U^Ty \in \mathbb R^p$. $\lambda \geq 0$ means $0 \leq \frac{d_i^2}{(d_i^2 + \lambda)^2}\leq 1$ which reflects the shrinkage nature of this.

$U$ gives an orthonormal basis for $\text{ColSpace}(X)$ so $z = U^Ty$ is the $p$ coordinates of $y$ expressed w.r.t. this basis (if $\text{rank}(X) < p$ then some singular values will be zero so it won't matter what values $z$ has for those coordinates). If $p > n$ then $U$ may be $n \times n$ in size instead but the ideas are the same.

So in light of this, $\|\hat\beta\|^2$ will be relatively large when $z$ is concentrated in the first few coordinates, so equivalently when $y$ can be well approximated by a linear combination of just the elements of $U$ with relatively large singular values. These are the elements that are used for giving the best low rank approximation of $X$ in the Frobenius sense (and other senses; this is just via the SVD) so if $y$ is well explained by just these then $y$ is close to the "essential" column space of $X$ rather than being noisier.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.