4
$\begingroup$

The derivation of the bias-variance tradeoff has been discussed pretty well here, see, e.g., https://stats.stackexchange.com/a/354284/46427.

I'm, however, skeptical of the existence of such a "tradeoff."

What the concept seems to be is this: the expected squared error can be reduced to three components: a bias component, a variance component, and an irreducible error. I have no problems with this.

But then we talk about this concept of bias and variance tradeoffs: i.e., among possible estimators, as bias goes down, variance goes up, and vice versa.

Doesn't this depend completely on the expected squared error being constant? Who's to say that if you have an estimator $\hat{f}$ of $Y = f(X) + \epsilon$ that you couldn't find an estimator $\hat{g}$ that has not only lowers expected squared error, but has lower bias and variance than $\hat{f}$ as well?

And because of this, I'm skeptical of the existence of such a tradeoff.

Change my mind. Explain to me why I'm wrong.

Improve this question
$\endgroup$
5
  • 4
    $\begingroup$ I don't think talk of a trade-off in any sense denies that one estimator might be better than another (subject to other limits). The prime context to me always seems to be one method, and tuning how it's applied. So, for example, a histogram can be tuned by varying bin width, and the results will vary accordingly. That doesn't rule out e.g. thinking about.a kernel density estimate in comparison. $\endgroup$ – Nick Cox Oct 11 '20 at 18:55
  • 3
    $\begingroup$ with regard to the claim that "among possible estimators, as bias goes down, variance goes up, and vice versa", it may be worth clarifying that this is usually not a statement about all possible estimators. it is more often the case that one would be considering some restricted family of estimators, for which there is some natural notion of complexity / capacity, and that within such a family, there will (often) be a tradeoff between bias and variance. it will also (often) be the case that a specific estimator within that family minimises the expected squared error. $\endgroup$ – πr8 Oct 11 '20 at 18:57
  • 3
    $\begingroup$ Many of the classical, textbook estimators have been proven to minimize quadratic loss ("expected squared error") among various classes of estimators, such as all linear ones. Thus, the answer to your challenge "who's to say" is the mathematicians. Among them are Gauss, Fisher, Neyman, Wald. $\endgroup$ – whuber♦ Oct 11 '20 at 19:06
  • 2
    $\begingroup$ Thank you all for your comments, @πr8, NickCox, Whuber. So it appears that the nuance I've missed is that this tradeoff isn't about estimators in general, and it's about tendencies (not necessarily holding in general, but tendencies) of estimators restricted to a certain family of estimators. It does make me wonder why I've never heard this aspect of the tradeoff explicitly called out, but the concept is more clear in my mind now. $\endgroup$ – Clarinetist Oct 11 '20 at 19:11
  • $\begingroup$ This bias/variance tradeoff idea is certainly over-hyped. I have never seen anyone using bias and variance as an evaluation metric in the development of any model. Seldom have I ever even seen a properly stated mathematical definition of it. When I do see it, it is with respect to $l^2$ loss which is basically never used in any application more sophisticated than linear regression. $\endgroup$ – Norman Dec 22 '20 at 0:18
4
$\begingroup$

First of all we have to say that bias-variance tradeoff (BVT) can be seen in respect not only of parameters estimators but also about prediction. Usually BVT is used in machine learning on prediction side and more precisely about the minimization of Expected Prediction Error (EPE). In this last sense the BVT was treated and derived in the discussion that you linked above.

Now you says:

Who's to say that if you have an estimator $\hat{f}$ of $Y = f(X) + \epsilon$ that you couldn't find an estimator $\hat{g}$ that has not only lowers expected squared error, but has lower bias and variance than $\hat{f}$ as well?

BVT do not exclude this possibility.

Usually in classical statistical or econometrics textbooks the focus is mainly on unbiased estimators (or consistent one, but the difference is not crucial here). So, what BVT tell you is that even if among all unbiased estimators you find the efficient one … remain possible that some biased ones achieve a lower $MSE$. I spoke about this possibility here (Mean squared error of OLS smaller than Ridge?), even if this answer was not appreciated much.

In general, if your goal is prediction, EPE minimization is the core, while in explanatory models the core is bias reduction. In math term you have to minimize two related but different loss functions, the tradeoff come from that. This discussion is about that: What is the relationship between minimizing prediciton error versus parameter estimation error?

Moreover what I said above is mainly related on linear models. While It seems me that in machine learning literature the concept BVT, the what that rendered it famous, is primarily related to the interpretability vs flexibility tradeoff. In general, the more flexible models have lower bias but higher variance. For less flexible models the opposite is true (lower variance and higher bias). Among the more flexible alternatives there are Neural Networks, among the less flexible there are linear regressions.

Doesn't this depend completely on the expected squared error being constant?

No. Among various alternative specifications (flexibility level) the test MSE (=EPE) is far from constant. Depend of the true model (true functional form), and the amount of data we have for training, we can find the flexibility level (specification) that permit us to achieve the EPE minimization.

This graph taken from: An Introduction to Statistical Learning with Applications in R - James Witten Hastie Tibshirani (pag 36)

enter image description here

gives us three examples. In the par 2.1.3 you can find a more exhaustive explanation of this last point.

Improve this answer
$\endgroup$
0
$\begingroup$

Who's to say that if you have an estimator $\hat{f}$ of $Y = f(X) + \epsilon$ that you couldn't find an estimator $\hat{g}$ that has not only lowers expected squared error, but has lower bias and variance than $\hat{f}$ as well?

A similar question was Bias / variance tradeoff math. In that question, it was asked if bias and variance could not be decreased simultaneously.

Often the starting-point is zero bias, and you can not lower the bias. So that's normally the trade-off, whether some alternative biased function will have lower variance and lower overall error than an unbiased function.

Sure if you have some bad estimator that has high bias and high variance, then there is no trade-off and you can make an improvement for both. But that is not the typical situation that you find in practice.

Normally you are considering a range of biased values and for each biased value, you have the situation that it has the most optimal variance possible for that biased value (at least the lowest that you know of, or the lowest that is practical to consider).

Below is the image of the linked question. It shows the bias-variance tradeoff for the bias of scaling the sample mean (as a predictor for the population mean). In the right image, the image is split in two.

  • If you are scaling with a factor above 1 then you have both an increased variance and increased bias. So that would indeed be silly. And when you have such a bad estimator, then there is no trade-off because you can make an improvement in both decreasing bias and decreasing variance.

  • If you are scaling with a factor below 1 then you do have a trade-off. Decreasing bias means increasing variance and vice versa.

    Within this particular set of biased estimators, you can say that you can't find an estimator that not only lowers variance, but also bias

    (Sure maybe you can find an even better estimator with a different type of bias. Indeed it may be difficult to proof that a particular biased estimator is the lowest variance estimator. Often, nobody is to say that it can't be improved).

overfitting and underfitting in shrinking of sample mean

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ I think the bias-variance tradeoff is best (i.e., most concretely) illustrated in the context of a regularization hyperparameter. An easily understood example is the ridge penalty on a linear model. Bias increases monotonically while variance decreases monotonically (in an asymptotic manner for both) as the ridge penalty is increased, but the minimum expected MSE is found at some nonzero value of the penalty. I think the "tradeoff" is best described as the need to balance the increasing bias with the decreasing variance in order to find this minimum expected error. $\endgroup$ – Josh Nov 25 '20 at 23:07
  • $\begingroup$ @Josh yes regularised regression is another good problem to showcase the principle. It is very similar to the case described in this answer. Both are sort of 'shrinking' the estimator magnitude which is decreasing the variance of the estimator. I personally prefer to view the trade-off with the shrinking of the parameters because the equations are easier. $\endgroup$ – Sextus Empiricus Nov 25 '20 at 23:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.