2
$\begingroup$

I am currently trying to understand the MSE of ridge regression. First, I am calculating the MSE mathematically, but I found it quite vague. After reviewing some books and articles I understood that

$$ \begin{aligned} \text{MSE}(\hat{\beta_R}) &= E[||\hat{\beta}_R-{\beta}||^2] \\ \Rightarrow\hat{\beta_{R}}-\beta &= ((X^TX+\lambda)^{-1}X^TX-I)\beta+e \\ \Rightarrow||\hat{\beta}_R-{\beta}||^2 &= (\hat{\beta_R}-{\beta})^T(\hat{\beta_R}-{\beta}) \end{aligned} $$

After that I got stuck because of the norm and expectation calculation. I tried to solve it, but it becomes so complicated.

I have checked books like: "The Elements of Statistical Learning" and "An Introduction to Statistical Learning".

Can anyone please clarify MSE of ridge regression or guide me to a good source?

$\endgroup$
2
  • 2
    $\begingroup$ Look at page 12 here. $\endgroup$ Jul 23, 2016 at 15:16
  • $\begingroup$ @Greenparker, Many thanks , actually I looked at this before, but it was not clear for me. $\endgroup$
    – jeza
    Jul 23, 2016 at 15:20

1 Answer 1

4
$\begingroup$

$\DeclareMathOperator{\rid}{\hat{\boldsymbol\beta}_\text{ridge}}\DeclareMathOperator{\ols}{\hat{\boldsymbol\beta}}\DeclareMathOperator{\bias}{\hat{\boldsymbol\beta^\ast}} \DeclareMathOperator{\tr} {trace}\DeclareMathOperator{\xx}{\mathbf X^\mathsf T\mathbf X}$

The good source would be straight from horse's mouth. (cf. $\rm [I]$).

$$\rid= \underbrace{\left[\xx + k\mathbf I\right]^{-1}}_{:=\mathbf W}\mathbf X^\mathsf T\mathbf y; \tag{1.I}$$ equivalently $$\rid =\underbrace{\left[\mathbf I +k\left(\xx\right)^{-1}\right]^{-1}}_{:=\mathbf Z}\ols.\tag{1.II}$$ As $\mathbf Z=\mathbf W\xx, $ $$\mathbf Z= \mathbf I-k\mathbf W. \tag 2$$

If $L^2(k):= \left(\rid-\boldsymbol\beta\right)^\mathsf T \left(\rid-\boldsymbol\beta\right), $ \begin{align}\mathbb E\left[L^2(k)\right]&= \mathbb E\left[\left(\ols-\boldsymbol\beta\right)^\mathsf T \mathbf Z^\mathsf T\mathbf Z\left(\ols-\boldsymbol\beta\right)\right]+ \left(\mathbf Z{\boldsymbol\beta}-\boldsymbol\beta\right)^\mathsf T \left(\mathbf Z{\boldsymbol\beta}-\boldsymbol\beta\right)\\ &= \mathbb E\left[{\boldsymbol\varepsilon}^\mathsf T\mathbf X\left(\xx\right)^{-1}\mathbf Z^\mathsf T\mathbf Z\left(\xx\right)^{-1}\mathbf X^\mathsf T\boldsymbol\varepsilon\right]+ \left(\mathbf Z{\boldsymbol\beta}-\boldsymbol\beta\right)^\mathsf T \left(\mathbf Z{\boldsymbol\beta}-\boldsymbol\beta\right)\\&= \sigma^2\tr\left[\left(\xx\right)^{-1}\mathbf Z^\mathsf T\mathbf Z\right]+ \boldsymbol\beta^\mathsf T\left(\mathbf Z-\mathbf I\right)^\mathsf T \left(\mathbf Z-\mathbf I\right) {\boldsymbol\beta}\\ &\overset{(2)}{=} \sigma^2\left[\tr\mathbf W-k\tr\mathbf W^2\right]+ k^2\boldsymbol\beta^\mathsf T\mathbf W^{2}\boldsymbol\beta\\ &= {\sigma^2\sum_{i=1}^p \frac{\lambda_i}{(\lambda_i + k)^2}}+ {k^2\sum_{i=1}^p \frac{\alpha_i^2}{(\lambda_i + k)^2}};\tag 3 \end{align} where $\boldsymbol\alpha = \mathbf P\boldsymbol\beta,~\mathbf P $ being the orthogonal matrix such that $\xx = \mathbf{ P\Lambda P}^\mathsf T, ~\mathbf\Lambda :=\operatorname{diag}(\lambda_i).$


Reference:

$\rm [I]$ Ridge Regression: Biased Estimation for Nonorthogonal Problems, Arthur E. Hoerl, Robert W. Kennard, Technometrics $42,$ no. $1~ (2000): ~80–86. $ https://doi.org/10.2307/1271436.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.