3
$\begingroup$

Suppose that $$y=f(x)+\epsilon$$ Where $\epsilon$ has mean $0$ and variance $\sigma^2_e$, independent of $x$.

Here is the composition of the mean-squared error into bias and variance:

$$\begin{align}\text{MSE} &=\mathbb E[(y-\hat f(x))^2]\\ &=(\mathbb E[\hat f(x)−f(x)])^2+\mathbb E[(\hat f(x)−\mathbb E[\hat f(x)])^2]+σ^2_e\\ &=\text{Bias }\quad\quad\quad\quad\quad\;\;+\text{Variance } \quad\quad\quad\quad\quad+ \text{ Irreducible Error}\end{align}$$

But here is an argument why there is no bias-variance tradeoff: Suppose we choose the correct function $\hat f=f$. Then

$\text{MSE}=\mathbb E[(y-f(x))^2]$. Using the law of total expectations this equals $\mathbb E\left[\mathbb E[(y-f(x))^2|x] \right]$. Now, because $\mathbb E[y|x]=\mathbb E[f(x)+\epsilon|x]=f(x)+0$, the expectation therefore equals $\mathbb E\left[\mathbb E[(y-\mathbb E[y|x])^2|x] \right]$=$\mathbb E[\mathbb E[(f(x)+\epsilon-f(x))^2|x]]$=$\mathbb E[\sigma^2_e]=\sigma^2_e$.

So by choosing $\hat f=f$, we have set both the bias and the variance to zero.

What is wrong with my argument?

$\endgroup$
  • 1
    $\begingroup$ Because the true $f$ may not be the $f$ you choose. In which case the variance is 0, but the bias can be arbitrarily large. The bias and the variance do not necessarily "trade off" as you can have biased and inefficient estimators. The point of the MSE is showing that the optimal estimator may not be unbiased. $\endgroup$ – AdamO Oct 10 '19 at 13:55
  • 1
    $\begingroup$ The problem is a an issue of searching for $f$. While you are searching unfortunately nobody tells you the value of MSE. The literature is a little confusing in the sense that MSE is a population property and is not readily available. What's usually available is the emprical risk which is a lower biased estimate of MSE. And when searching with the aid of emprical risk you don't end up with the true $f$. $\endgroup$ – Cagdas Ozgenc Oct 10 '19 at 13:58
  • $\begingroup$ @AdamO "The point of the MSE is showing that the optimal estimator may not be unbiased." I don't think this is technically very correct. There is surely a lucky estimator that finds an unbiased and 0 variance estimate. it is just almost impossible to find it. $\endgroup$ – Cagdas Ozgenc Oct 10 '19 at 14:14
7
$\begingroup$

Nothing is wrong with your argument. Since

  • by your design, the "reducible error" (a term suggested in this thread) equals zero AND
  • by definition, variance is nonnegative AND
  • by definition, squared bias is nonnegative,

we have that variance equals zero and squared bias equals zero. There can be no bias-variance trade-off in this setup. The trade-off only applies to setups where the true data generating process is unknown (which is true is most real life situations).

Moreover, the bias-variance trade-off does not imply all models will have the same reducible error (and hence the same $\text{MSE}$) while the proportions of squared bias and variance within it will vary. The reducible error will be different for different models, and if you happen to find an $\hat{f}$ such that $\hat{f}=f$, the reducible error will be zero.

Rather, the bias-variance trade-off says that there is no free lunch within a class of nested models: pursuing low bias requires increasing complexity which increases variance, and pursuing low variance requires decreasing complexity which increases bias. However, moving across nonnested models, you may luck out in decreasing bias and variance simultaneously.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ I think the reason I'm confused is because I don't know what the expectations are being taken over. Are they being taken over $x$? (which I should interpret as an element of the test data?), or over $\hat f$, as a function of the training data? Is there a very formal explanation of the bias variance tradeoff, which includes all the basic context (the fact that $\hat f$ is a random function, that there is a distinction between test and training data, etc)? $\endgroup$ – user56834 Oct 10 '19 at 13:58
  • $\begingroup$ @user56834, good questions! I will try to get back to you in a while. $\endgroup$ – Richard Hardy Oct 10 '19 at 15:00
  • $\begingroup$ I understand if you are busy, but just writing this as a reminder :) $\endgroup$ – user56834 Jun 5 at 4:15
  • $\begingroup$ @user56834, sorry, I forgot to get back to you. It is difficult to find time to think deeply enough, and this is one of the threads that requires exactly that from me. Perhaps you could post a new question asking precisely what the expectations are taken over, leaving your paradox aside (just linking to this thread)? I would appreciate if you pinged me after having posted the question. I am very much interested in an answer there. Also, here is a somewhat relevant link: Rob J. Hyndman "Bias variance decomposition". $\endgroup$ – Richard Hardy Jun 5 at 11:06
  • $\begingroup$ @user56834, I have posted the question myself: "Rigorous statement of expectations for the bias-variance trade-off". $\endgroup$ – Richard Hardy Jun 9 at 4:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.