Mean Squared Error (MSE) of Ridge Regression

Question

I am currently trying to understand the MSE of ridge regression. First, I am calculating the MSE mathematically, but I found it quite vague. After reviewing some books and articles I understood that

$$ \begin{aligned} \text{MSE}(\hat{\beta_R}) &= E[||\hat{\beta}_R-{\beta}||^2] \\ \Rightarrow\hat{\beta_{R}}-\beta &= ((X^TX+\lambda)^{-1}X^TX-I)\beta+e \\ \Rightarrow||\hat{\beta}_R-{\beta}||^2 &= (\hat{\beta_R}-{\beta})^T(\hat{\beta_R}-{\beta}) \end{aligned} $$

After that I got stuck because of the norm and expectation calculation. I tried to solve it, but it becomes so complicated.

I have checked books like: "The Elements of Statistical Learning" and "An Introduction to Statistical Learning".

Can anyone please clarify MSE of ridge regression or guide me to a good source?

Answer 1

$\DeclareMathOperator{\rid}{\hat{\boldsymbol\beta}_\text{ridge}}\DeclareMathOperator{\ols}{\hat{\boldsymbol\beta}}\DeclareMathOperator{\bias}{\hat{\boldsymbol\beta^\ast}} \DeclareMathOperator{\tr} {trace}\DeclareMathOperator{\xx}{\mathbf X^\mathsf T\mathbf X}$

The good source would be straight from horse's mouth. (cf. $\rm [I]$).

$$\rid= \underbrace{\left[\xx + k\mathbf I\right]^{-1}}_{:=\mathbf W}\mathbf X^\mathsf T\mathbf y; \tag{1.I}$$ equivalently $$\rid =\underbrace{\left[\mathbf I +k\left(\xx\right)^{-1}\right]^{-1}}_{:=\mathbf Z}\ols.\tag{1.II}$$ As $\mathbf Z=\mathbf W\xx, $ $$\mathbf Z= \mathbf I-k\mathbf W. \tag 2$$

If $L^2(k):= \left(\rid-\boldsymbol\beta\right)^\mathsf T \left(\rid-\boldsymbol\beta\right), $ \begin{align}\mathbb E\left[L^2(k)\right]&= \mathbb E\left[\left(\ols-\boldsymbol\beta\right)^\mathsf T \mathbf Z^\mathsf T\mathbf Z\left(\ols-\boldsymbol\beta\right)\right]+ \left(\mathbf Z{\boldsymbol\beta}-\boldsymbol\beta\right)^\mathsf T \left(\mathbf Z{\boldsymbol\beta}-\boldsymbol\beta\right)\\ &= \mathbb E\left[{\boldsymbol\varepsilon}^\mathsf T\mathbf X\left(\xx\right)^{-1}\mathbf Z^\mathsf T\mathbf Z\left(\xx\right)^{-1}\mathbf X^\mathsf T\boldsymbol\varepsilon\right]+ \left(\mathbf Z{\boldsymbol\beta}-\boldsymbol\beta\right)^\mathsf T \left(\mathbf Z{\boldsymbol\beta}-\boldsymbol\beta\right)\\&= \sigma^2\tr\left[\left(\xx\right)^{-1}\mathbf Z^\mathsf T\mathbf Z\right]+ \boldsymbol\beta^\mathsf T\left(\mathbf Z-\mathbf I\right)^\mathsf T \left(\mathbf Z-\mathbf I\right) {\boldsymbol\beta}\\ &\overset{(2)}{=} \sigma^2\left[\tr\mathbf W-k\tr\mathbf W^2\right]+ k^2\boldsymbol\beta^\mathsf T\mathbf W^{2}\boldsymbol\beta\\ &= {\sigma^2\sum_{i=1}^p \frac{\lambda_i}{(\lambda_i + k)^2}}+ {k^2\sum_{i=1}^p \frac{\alpha_i^2}{(\lambda_i + k)^2}};\tag 3 \end{align} where $\boldsymbol\alpha = \mathbf P\boldsymbol\beta,~\mathbf P $ being the orthogonal matrix such that $\xx = \mathbf{ P\Lambda P}^\mathsf T, ~\mathbf\Lambda :=\operatorname{diag}(\lambda_i).$

---

Reference:

$\rm [I]$ Ridge Regression: Biased Estimation for Nonorthogonal Problems, Arthur E. Hoerl, Robert W. Kennard, Technometrics $42,$ no. $1~ (2000): ~80–86. $ https://doi.org/10.2307/1271436.