Why is there a bias variance tradeoff? A counterexample

Question

Suppose that $$y=f(x)+\epsilon$$ Where $\epsilon$ has mean $0$ and variance $\sigma^2_e$, independent of $x$.

Here is the composition of the mean-squared error into bias and variance:

$$\begin{align}\text{MSE} &=\mathbb E[(y-\hat f(x))^2]\\ &=(\mathbb E[\hat f(x)−f(x)])^2+\mathbb E[(\hat f(x)−\mathbb E[\hat f(x)])^2]+σ^2_e\\ &=\text{Bias }\quad\quad\quad\quad\quad\;\;+\text{Variance } \quad\quad\quad\quad\quad+ \text{ Irreducible Error}\end{align}$$

But here is an argument why there is no bias-variance tradeoff: Suppose we choose the correct function $\hat f=f$. Then

$\text{MSE}=\mathbb E[(y-f(x))^2]$. Using the law of total expectations this equals $\mathbb E\left[\mathbb E[(y-f(x))^2|x] \right]$. Now, because $\mathbb E[y|x]=\mathbb E[f(x)+\epsilon|x]=f(x)+0$, the expectation therefore equals $\mathbb E\left[\mathbb E[(y-\mathbb E[y|x])^2|x] \right]$=$\mathbb E[\mathbb E[(f(x)+\epsilon-f(x))^2|x]]$=$\mathbb E[\sigma^2_e]=\sigma^2_e$.

So by choosing $\hat f=f$, we have set both the bias and the variance to zero.

What is wrong with my argument?

Answer 1

Nothing is wrong with your argument. Since

• by your design, the "reducible error" (a term suggested in this thread) equals zero AND
• by definition, variance is nonnegative AND
• by definition, squared bias is nonnegative,

we have that variance equals zero and squared bias equals zero. There can be no bias-variance trade-off in this setup. The trade-off only applies to setups where the true data generating process is unknown (which is true is most real life situations).

Moreover, the bias-variance trade-off does not imply all models will have the same reducible error (and hence the same $\text{MSE}$) while the proportions of squared bias and variance within it will vary. The reducible error will be different for different models, and if you happen to find an $\hat{f}$ such that $\hat{f}=f$, the reducible error will be zero.

Rather, the bias-variance trade-off says that there is no free lunch within a class of nested models: pursuing low bias requires increasing complexity which increases variance, and pursuing low variance requires decreasing complexity which increases bias. However, moving across nonnested models, you may luck out in decreasing bias and variance simultaneously.