Return to Answer

The original source of this answer can be found in my regression course notes. This version corrects some minor errors.

Let the columns of $X$ be standardized, as well as $y$ itself. For $\lambda \gt 0$ the ridge estimator is $$\begin{aligned}\hat\beta_R &= (X^\prime X + \lambda)^{-1}X^\prime y \\ &= (VD^2V^\prime + \lambda\,1_p)^{-1}VDU^\prime y \\ &= (VD^2V^\prime + \lambda V V^\prime)^{-1}VDU^\prime y \\ &= (V(D^2 + \lambda)V^\prime)^{-1} VDU^\prime y \\ &= V(D^2+\lambda)^{-1}V^\prime V DU^\prime y \\ &= V(D^2 + \lambda)^{-1} D U^\prime y.\end{aligned}$$

To avoid distractions, the case of one of more singular values was excluded in this discussion. In such circumstances, if we conventionally take "$d_{ii}^{-1}$" to be zero, then everything still works. This is what is going on when generalized inverses are used to solve the Normal equations.

Let the columns of $X$ be standardized, as well as $y$ itself. (This means we no longer need a constant column in $X$.) For $\lambda \gt 0$ the ridge estimator is $$\begin{aligned}\hat\beta_R &= (X^\prime X + \lambda)^{-1}X^\prime y \\ &= (VD^2V^\prime + \lambda\,1_p)^{-1}VDU^\prime y \\ &= (VD^2V^\prime + \lambda V V^\prime)^{-1}VDU^\prime y \\ &= (V(D^2 + \lambda)V^\prime)^{-1} VDU^\prime y \\ &= V(D^2+\lambda)^{-1}V^\prime V DU^\prime y \\ &= V(D^2 + \lambda)^{-1} D U^\prime y.\end{aligned}$$

To avoid distractions, the case of one of more zero singular values was excluded in this discussion. In such circumstances, if we conventionally take "$d_{ii}^{-1}$" to be zero, then everything still works. This is what is going on when generalized inverses are used to solve the Normal equations.

It can simplify formulas. Here are some examples.

Consider the regression $y = X\beta + \varepsilon$ where, as usual, the $\varepsilon$ are independent and identically distributed according to a law that has zero expectation and finite variance $\sigma^2$. Recall the least squares solution via the Normal Equations, $$\hat\beta = (X^\prime X)^{-1} X^\prime y.$$ Use the SVD:

The only difference between this and $X^\prime = VDU^\prime$ is that the reciprocals of the elements of $D$ are used!

It can simplify formulas. This works both algebraically and conceptually. Here are some examples.

Consider the regression $y = X\beta + \varepsilon$ where, as usual, the $\varepsilon$ are independent and identically distributed according to a law that has zero expectation and finite variance $\sigma^2$. The least squares solution via the Normal Equations is $$\hat\beta = (X^\prime X)^{-1} X^\prime y.$$ Applying the SVD and simplifying the resulting algebraic mess (which is easy) provides a nice insight:

The only difference between this and $X^\prime = VDU^\prime$ is that the reciprocals of the elements of $D$ are used! In other words, the "equation" $y=X\beta$ is solved by "inverting" $X$: this pseudo-inversion undoes the rotations $U$ and $V^\prime$ (merely by transposing them) and undoes the multiplication (represented by $D$) separately in each principal direction.