Decompose ridge regression bias error into model bias and estimation bias How can I show that the in-sample bias error in Ridge regression can be decomposed into model bias plus estimation bias?  I.e., if $Avg$ takes the average over all the input variables $x$ in the sample, and $E$ denotes the in-sample expectation value over the noise component in the underlying modeled function $f$, then:
$$
Avg[(f(x)-E[x'\beta_{Ridge}])^2] = Avg[(f(x)-x'\beta_{OLS})^2] + 
Avg[(x'\beta_{OLS}-E[x'\beta_{Ridge}])^2] 
$$
$$
= Squared Model Bias  +  Squared Estimation Bias
$$
This is stated in Elements of Statistical Learning, Hastie et al, p.224, result 7.14
 A: It turns out this can be resolved by rewriting as matrices and using the optimality condition for $\beta_{OLS}$.  Rewriting as matrices, taking the expectation value over $x'\beta_{Ridge}$ (this just results in dropping the error term $\epsilon$ in $y+\epsilon$), dropping the subscript $OLS$, and using subscript $R$ instead of $Ridge$, the above expression can be re-written:
$$
\frac{1}{N}[(y-X\beta)^2+(X\beta-X\beta_R)^2]
$$
Dropping $N$ and using the optimality condition for $\beta$, i.e. $X'X\beta=X'y$, the first term can be re-written:
$$
y'y-y'X\beta
$$
Using the "hat" matrix $H = X(X'X)^{-1}X'$, this can be written:
$$
y'y-y'Hy \quad \text{(eq. 1)}\\
$$
The $SquaredEstimationBias$ can be further simplified using the $H$ matrix and the corresponding matrix for Ridge regression, $X(X'X+\lambda I)^{-1}X'$, let us call it $H_R$. One can check that these matrices have the following nice properties, which have to do with the fact that $H$ is a projection matrix: 
$$
H'H = H
$$
$$
H'H_R = H_R'H= H_R
$$
Now, if we write out the $Squared Estimation Bias$ term using the "hat" matrices and their properties, we arrive at:
$$
y'Hy-2y'H_Ry+y'H_R'H_Ry
$$
The first of the terms directly above cancels the second term in eq. 1 for the $Squared Model Bias$, $(-y'Hy)$, leaving a total sum for $Squared Model Bias + Squared Estimation Bias$ of:
$$
y'y-2y'H_Ry+y'H_R'H_Ry
$$
Equating $y$ with $f(x)$ (this is implied by the taking of expectations) this can be re-written to match the total bias error, as required:
$$
(y-X\beta_R)^2 = N*Avg(f(x)-x'\beta_R)^2=N*Avg(f(x)-E[x'\beta_R])^2
$$
Given the properties that we have used here, I believe this result holds for any model that can be written as $X\hat{\beta}$, a linear combination of the input variables, not just Ridge regression.
