Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
I am going through this blog post to understand the Variance - Bias Tradeoff. In this, the author mentions:
Err(x) = E[(Y - f'(x))^2]
I understand this. But then, she decomposes it as:
Err(x) = (E([f'(x)] - f(x))^2 + E[(f(x) - E[f'(x)])^2]
I do not understand where this came from. I am aware that E(X^2) = Var(X) + (E[X])^2. But how does that lead to the above expression?
You've a typo. In the second summand, it should be $\hat f(x)$. Substitute $y=f(x)+\epsilon$, and assuming $x$ is constant: $$\begin{align}\mathbb E[(y-\hat f(x))^2]&=\mathbb E[(f(x)+\epsilon-\hat f(x))^2]\\&=\mathbb E[(f(x)-\hat f(x))^2]+\mathbb E[\epsilon^2]+\overbrace{2\mathbb E[\epsilon (f(x)-\hat f(x))]}^0\\&=\mathbb E[(f(x)-\mathbb E[\hat f(x)]-(\hat f(x)-\mathbb E[\hat f(x)]))^2]+\sigma_\epsilon^2\\&=\mathbb E[(f(x)-\mathbb E[\hat f (x)])^2]+\mathbb E[(\hat f(x)-\mathbb E[\hat f(x)])^2]\\&\ \ \ \ \ \ \ \ -\overbrace{2\mathbb E[(f(x)-\mathbb E[\hat f(x)])(\hat f(x)-\mathbb E[\hat f(x)])]}^{0}+\sigma_\epsilon^2\\&=\mathbb E[(f(x)-\mathbb E[\hat f (x)])^2]+\mathbb E[(\hat f(x)-\mathbb E[\hat f(x)])^2]+\sigma_\epsilon^2\\&= (\mathbb E[\hat f (x)]-f(x))^2+\mathbb E[(\hat f(x)-\mathbb E[\hat f(x)])^2]+\sigma_\epsilon^2 \end{align}$$
The first $0$ is simple (with the typical assumption that error is orthogonal to predicted $y$, i.e. $\hat y=\hat f(x)$): $$\mathbb E[\epsilon (f(x)-\hat f(x))]=\mathbb E[\epsilon]\mathbb E[f(x)]-\overbrace{\mathbb E[\epsilon\hat f(x)]}^0=0$$
The second one is:
$$\begin{align}\mathbb E[(f(x)-\mathbb E[\hat f(x)])(\hat f(x)-\mathbb E[\hat f(x)])]&=\mathbb E[f(x)-\mathbb E[\hat f(x)]]\mathbb E[\hat f(x)-\mathbb E[\hat f(x)]]\\&=(f(x)-\mathbb E[\hat f(x)])\underbrace{(\mathbb E[\hat f(x)]-\mathbb E[\hat f(x)])}_0\\&=0\end{align}$$
