The question appears to ask for a demonstration that Ridge Regression shrinks coefficient estimates towards zero, using a spectral decomposition. Ultimately, the spectral decomposition derives from the Singular Value Decomposition (SVD). Therefore, this post starts with SVD. It explains it in simple terms and then illustrates it with important applications. Then it provides the requested (algebraic) demonstration. (The algebra, of course, is identical to the geometric demonstration; it merely is couched in a different language.)
The original source of this answer can be found in my regression course notes.
##What the SVD is
Any $n\times p$ matrix $X$, with $p \le n$, can be written $$X = UDV^\prime$$ where
$U$ is an $n\times p$ matrix.
- The columns of $U$ have length $1$.
- The columns of $U$ are mutually orthogonal.
- They are called the principal components of $X$.
$V$ is a $p \times p$ matrix.
- The columns of $V$ have length $1$.
- The columns of $V$ are mutually orthogonal.
- This makes $V$ a rotation of $\mathbb{R}^p$.
$D$ is a diagonal $p \times p$ matrix.
- The diagonal elements $d_{11}, d_{22}, \ldots, d_{pp}$ are not negative. These are the singular values of $X$.
- If we wish, we may order them from largest to smallest.
Criteria (1) and (2) assert that both $U$ and $V$ are orthonormal matrices. They can be neatly summarized by the conditions
$$U^\prime U = 1_n,\ V^\prime V = 1_p.$$
##What it does for us
It can simplify formulas. Here are some examples.
###The Normal Equations
Consider the regression $y = X\beta + \varepsilon$. Recall the least squares solution via the Normal Equations, $$\hat\beta = (X^\prime X)^{-1} X^\prime y.$$ Use the SVD:
$$(X^\prime X)^{-1} X^\prime = ((UDV^\prime)^\prime (UDV^\prime))^{-1} (UDV^\prime)^\prime \\= (VDU^\prime U D V^\prime)^{-1} (VDU^\prime) = VD^{-2}V^\prime VDU^\prime = VD^{-1}U^\prime.$$
The only difference between this and $X^\prime = VDU^\prime$ is that the reciprocals of the elements of $D$ are used!
###Covariance of the coefficient estimates
Recall that the covariance of the estimates is $$\text{Cov}(\hat\beta) = \sigma^2(X^\prime X)^{-1}.$$ Using the SVD, this becomes $$\sigma^2(V D^2 V^\prime)^{-1} = \sigma^2 V D^{-2} V^\prime.$$ In other words, the covariance acts like that of $k$ orthogonal variables, each with variances $d^2_{ii}$, that have been rotated in $\mathbb{R}^k$.
###The Hat matrix
The hat matrix is $$H = X(X^\prime X)^{-1} X^\prime.$$ By means of the preceding result we may rewrite it as $$H = (UDV^\prime)(VD^{-1}U^\prime) = UU^\prime.$$ Simple!
###Eigenanalysis (spectral decomposition)
Since $$X^\prime X = VDU^\prime U D V^\prime = VD^2V^\prime$$ and $$XX^\prime = UDV^\prime VDU^\prime = UD^2U^\prime,$$ it is immediate that
- The eigenvalues of $X^\prime X$ and $XX^\prime$ are the squares of the singular values.
- The columns of $V$ are the eigenvectors of $X^\prime X$.
- The columns of $U$ are some of the eigenvectors of $X X^\prime$. (Other eigenvectors exist but correspond to zero eigenvalues.)
SVD can diagnose and solve collinearity problems.
###Approximating the regressors
When you replace the smallest singular values with zeros, you will change the product $UDV^\prime$ only slightly. Now, however, the zeros eliminate the corresponding columns of $U$, effectively reducing the number of variables. Provided those eliminated columns have little correlation with $y$, this can work effectively as a variable-reduction technique.
##Ridge Regression
Let the columns of $X$ be standardized, as well as $y$ itself. For $\lambda \gt 0$ the ridge estimator is $$\begin{aligned}\hat\beta_R &= (X^\prime X + \lambda)^{-1}X^\prime y \\ &= (VD^2V^\prime + \lambda)^{-1}VDU^\prime y \\ &= (VD^2V^\prime + \lambda V^\prime V)^{-1}VDU^\prime y \\ &= (V(D^2 + \lambda)V^\prime)^\prime VDU^\prime y \\ &= V(D^2+\lambda)^{-1}V^\prime V DU^\prime y \\ &= V(D^2 + \lambda)^{-1} D U^\prime y.\end{aligned}$$
The difference between this and $\hat\beta$ is the replacement of $D^{-1} = D^{-2}D$ by $(D^2+\lambda)^{-1}D$. Because (when $\lambda \gt 0$) the denominator is obviously greater than the numerator, they "shrink towards zero." (At the same time the covariances of the estimates change from $d^2_{ii}$ to $d^2_{ii} / (d^2_{ii} + \lambda)^2$: they, too grow smaller. They are not necessarily better, though: unlike $\hat\beta$, the ridge regression estimates are biased and the bias grows with $\lambda$.)