Is the idea of a bias-variance "tradeoff" a false construct?

Question

The derivation of the bias-variance tradeoff has been discussed pretty well here, see, e.g., https://stats.stackexchange.com/a/354284/46427.

I'm, however, skeptical of the existence of such a "tradeoff."

What the concept seems to be is this: the expected squared error can be reduced to three components: a bias component, a variance component, and an irreducible error. I have no problems with this.

But then we talk about this concept of bias and variance tradeoffs: i.e., among possible estimators, as bias goes down, variance goes up, and vice versa.

Doesn't this depend completely on the expected squared error being constant? Who's to say that if you have an estimator $\hat{f}$ of $Y = f(X) + \epsilon$ that you couldn't find an estimator $\hat{g}$ that has not only lowers expected squared error, but has lower bias and variance than $\hat{f}$ as well?

And because of this, I'm skeptical of the existence of such a tradeoff.

Change my mind. Explain to me why I'm wrong.

Answer 1

First of all we have to say that bias-variance tradeoff (BVT) can be seen in respect not only of parameters estimators but also about prediction. Usually BVT is used in machine learning on prediction side and more precisely about the minimization of Expected Prediction Error (EPE). In this last sense the BVT was treated and derived in the discussion that you linked above.

Now you says:

Who's to say that if you have an estimator $\hat{f}$ of $Y = f(X) + \epsilon$ that you couldn't find an estimator $\hat{g}$ that has not only lowers expected squared error, but has lower bias and variance than $\hat{f}$ as well?

BVT do not exclude this possibility.

Usually in classical statistical or econometrics textbooks the focus is mainly on unbiased estimators (or consistent one, but the difference is not crucial here). So, what BVT tell you is that even if among all unbiased estimators you find the efficient one … remain possible that some biased ones achieve a lower $MSE$. I spoke about this possibility here (Mean squared error of OLS smaller than Ridge?), even if this answer was not appreciated much.

In general, if your goal is prediction, EPE minimization is the core, while in explanatory models the core is bias reduction. In math term you have to minimize two related but different loss functions, the tradeoff come from that. This discussion is about that: What is the relationship between minimizing prediciton error versus parameter estimation error?

Moreover what I said above is mainly related on linear models. While It seems me that in machine learning literature the concept BVT, the what that rendered it famous, is primarily related to the interpretability vs flexibility tradeoff. In general, the more flexible models have lower bias but higher variance. For less flexible models the opposite is true (lower variance and higher bias). Among the more flexible alternatives there are Neural Networks, among the less flexible there are linear regressions.

Doesn't this depend completely on the expected squared error being constant?

No. Among various alternative specifications (flexibility level) the test MSE (=EPE) is far from constant. Depend of the true model (true functional form), and the amount of data we have for training, we can find the flexibility level (specification) that permit us to achieve the EPE minimization.

This graph taken from: An Introduction to Statistical Learning with Applications in R - James Witten Hastie Tibshirani (pag 36)

gives us three examples. In the par 2.1.3 you can find a more exhaustive explanation of this last point.

Answer 2

Who's to say that if you have an estimator $\hat{f}$ of $Y = f(X) + \epsilon$ that you couldn't find an estimator $\hat{g}$ that has not only lowers expected squared error, but has lower bias and variance than $\hat{f}$ as well?

A similar question was Bias / variance tradeoff math. In that question, it was asked if bias and variance could not be decreased simultaneously.

---

Often the starting-point is zero bias, and you can not lower the bias. So that's normally the trade-off, whether some alternative biased function will have lower variance and lower overall error than an unbiased function.

Sure if you have some bad estimator that has high bias and high variance, then there is no trade-off and you can make an improvement for both. But that is not the typical situation that you find in practice.

Normally you are considering a range of biased values and for each biased value, you have the situation that it has the most optimal variance possible for that biased value (at least the lowest that you know of, or the lowest that is practical to consider).

Below is the image of the linked question. It shows the bias-variance tradeoff for the bias of scaling the sample mean (as a predictor for the population mean). In the right image, the image is split in two.

• If you are scaling with a factor above 1 then you have both an increased variance and increased bias. So that would indeed be silly. And when you have such a bad estimator, then there is no trade-off because you can make an improvement in both decreasing bias and decreasing variance.

• If you are scaling with a factor below 1 then you do have a trade-off. Decreasing bias means increasing variance and vice versa.

  Within this particular set of biased estimators, you can say that you can't find an estimator that not only lowers variance, but also bias

  (Sure maybe you can find an even better estimator with a different type of bias. Indeed it may be difficult to proof that a particular biased estimator is the lowest variance estimator. Often, nobody is to say that it can't be improved).