Trace of the Hat Matrix in Ridge Regression

Question

Generally, I know that the trace of the hat matrix ($H$) is equal to the rank of H since it is an orthogonal projection. If I wanted to show the trace of $H$ in ridge regression, would I be able to somehow prove that it equals the degrees of freedom?

Answer 1

See What are the leverage values for Ridge regression? for a related question. The hat matrix for ridge regression (with notation from that question, $X_0^T X_0 =\lambda I$) can be written $$ H_\lambda = X \left( X^T X +\lambda I\right)^{-1} X^T $$ and its trace will be lesser than the dimension of the column space of $\left(\begin{smallmatrix} X \\ \sqrt{\lambda} I \end{smallmatrix}\right)$, since the usual hat matrix from the augmented variable representation is $$ \begin{pmatrix} X \left( X^T X +\lambda I\right)^{-1} X^T & \cdot \\ \cdot & \sqrt\lambda I \left( X^T X +\lambda I\right)^{-1} \sqrt\lambda I \end{pmatrix} $$ ($\cdot$ parts left out because irrelevant for the trace.) The first diagonal block is the hat matrix for ridge regression (which is not a projector anymore, it has eigenvalues less than one.) The full matrix is the hat matrix for OLS in the augmented data representation for ridge, so its trace is the dimension of the column space of $\left(\begin{smallmatrix} X \\ \sqrt{\lambda} I \end{smallmatrix}\right)$. That way the trace of the second diagonal block is a measure of the smoothing effect introduced by ridge, and the trace of the forst diagonal block is the usual definition of the effective number of degrees of freedom for ridge regression.