# norm of ridge regression estimator

is there a characterization or an upper bound on the norm of the ridge regression estimator (coefficients)? As the Tikhonov regularization attempts to regularize these coefficients as part of the loss function, is there an exact form for this norm after the estimator is computed?

• There is a closed form solution to the ridge regression problem, hence a closed form expression for the norm. Jan 8 at 6:52

We have $$\hat\beta = (X^TX + \lambda I)^{-1}X^Ty$$ with $$X \in \mathbb R^{n\times p}$$ so, as @passerby51 commented, $$\|\hat\beta\|^2 = y^TX(X^TX + \lambda I)^{-2}X^Ty.$$ To interpret this, let $$X = UDV^T$$ be the SVD of $$X$$. Then $$\|\hat\beta\|^2 = y^T UD(D^2 + \lambda I)^{-2}DU^Ty = \sum_{i=1}^p \frac{d_i^2}{(d_i^2 + \lambda)^2} z_i^2$$ with $$z = U^Ty \in \mathbb R^p$$. $$\lambda \geq 0$$ means $$0 \leq \frac{d_i^2}{(d_i^2 + \lambda)^2}\leq 1$$ which reflects the shrinkage nature of this.
$$U$$ gives an orthonormal basis for $$\text{ColSpace}(X)$$ so $$z = U^Ty$$ is the $$p$$ coordinates of $$y$$ expressed w.r.t. this basis (if $$\text{rank}(X) < p$$ then some singular values will be zero so it won't matter what values $$z$$ has for those coordinates). If $$p > n$$ then $$U$$ may be $$n \times n$$ in size instead but the ideas are the same.
So in light of this, $$\|\hat\beta\|^2$$ will be relatively large when $$z$$ is concentrated in the first few coordinates, so equivalently when $$y$$ can be well approximated by a linear combination of just the elements of $$U$$ with relatively large singular values. These are the elements that are used for giving the best low rank approximation of $$X$$ in the Frobenius sense (and other senses; this is just via the SVD) so if $$y$$ is well explained by just these then $$y$$ is close to the "essential" column space of $$X$$ rather than being noisier.