1
$\begingroup$

If we consider the variance bias trade off equation, stated as:

$$\newcommand{\Var}{{\rm Var}} E(y_0-\hat{f}(x_0))^2=\Var(\hat{f}(x_0))+({\rm Bias}(\hat{f}(x_0)))^2+\Var(\varepsilon) $$

We assume that the model is $y=f(x)+\varepsilon$ and our test data is $(x_0,y_0)$.

If we assume we are using the correct model then the bias should be zero. Further, $E(\hat{f}(x_0))=y_0$ too, which leads to the left hand term of the original equation being equal to $\Var(\hat{f}(x_0))$. Putting all this together into the original equation leads us to find that $\Var(\varepsilon)=0$ which is clearly false by design.

Clearly there is something wrong with my understanding. Which step(s) of the above are therefore false?

$\endgroup$
1
$\begingroup$

The squared term on the left hand side should be inside the expectation. The equation should be: $$ E[(y_0 - \hat{f}(x_0))^2] = Var(\hat{f}(x_0)) + Bias(\hat{f}(x_0))^2 + Var(\epsilon)$$

Therefore you can't use linearity of expectation. If your equation was true, the LHS would actually be 0 for an unbiased $\hat{f}$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.