How can I show that the in-sample bias error in Ridge regression can be decomposed into model bias plus estimation bias? I.e., if $Avg$ takes the average over all the input variables $x$ in the sample, and $E$ denotes the in-sample expectation value over the noise component in the underlying modeled function $f$, then:

$$ Avg[(f(x)-E[x'\beta_{Ridge}])^2] = Avg[(f(x)-x'\beta_{OLS})^2] + Avg[(x'\beta_{OLS}-E[x'\beta_{Ridge}])^2] $$ $$ = Squared Model Bias + Squared Estimation Bias $$ This is stated in Elements of Statistical Learning, Hastie et al, p.224, result 7.14


1 Answer 1


It turns out this can be resolved by rewriting as matrices and using the optimality condition for $\beta_{OLS}$. Rewriting as matrices, taking the expectation value over $x'\beta_{Ridge}$ (this just results in dropping the error term $\epsilon$ in $y+\epsilon$), dropping the subscript $OLS$, and using subscript $R$ instead of $Ridge$, the above expression can be re-written:

$$ \frac{1}{N}[(y-X\beta)^2+(X\beta-X\beta_R)^2] $$

Dropping $N$ and using the optimality condition for $\beta$, i.e. $X'X\beta=X'y$, the first term can be re-written:

$$ y'y-y'X\beta $$

Using the "hat" matrix $H = X(X'X)^{-1}X'$, this can be written:

$$ y'y-y'Hy \quad \text{(eq. 1)}\\ $$

The $SquaredEstimationBias$ can be further simplified using the $H$ matrix and the corresponding matrix for Ridge regression, $X(X'X+\lambda I)^{-1}X'$, let us call it $H_R$. One can check that these matrices have the following nice properties, which have to do with the fact that $H$ is a projection matrix:

$$ H'H = H $$ $$ H'H_R = H_R'H= H_R $$

Now, if we write out the $Squared Estimation Bias$ term using the "hat" matrices and their properties, we arrive at:

$$ y'Hy-2y'H_Ry+y'H_R'H_Ry $$

The first of the terms directly above cancels the second term in eq. 1 for the $Squared Model Bias$, $(-y'Hy)$, leaving a total sum for $Squared Model Bias + Squared Estimation Bias$ of:

$$ y'y-2y'H_Ry+y'H_R'H_Ry $$

Equating $y$ with $f(x)$ (this is implied by the taking of expectations) this can be re-written to match the total bias error, as required:

$$ (y-X\beta_R)^2 = N*Avg(f(x)-x'\beta_R)^2=N*Avg(f(x)-E[x'\beta_R])^2 $$

Given the properties that we have used here, I believe this result holds for any model that can be written as $X\hat{\beta}$, a linear combination of the input variables, not just Ridge regression.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.