3
$\begingroup$

In machine learning, high bias prevents a model from properly fitting even the training set.

So, does a model have zero training error if and only if it has zero bias?

$\endgroup$
6
  • $\begingroup$ I'm rethinking my earlier example $\endgroup$
    – astel
    Jul 14, 2020 at 6:57
  • $\begingroup$ @astel No problem. Decision trees might be a hard example to work with since they're non-parametric (and I'm not sure how bias is defined in that case). $\endgroup$
    – kennysong
    Jul 14, 2020 at 7:10
  • $\begingroup$ Trivial counterexample: say that your "model" is function(x) return(0) and you are using it with train data that has only 0 values as labels, so your model is clearly biased (makes predictions independent of data), but the train error would be 0. $\endgroup$
    – Tim
    Jul 14, 2020 at 8:40
  • $\begingroup$ @Tim Ah true! A high bias model can have zero training error by chance. What about the reverse direction? If a model has no bias, then will the training error be zero? $\endgroup$
    – kennysong
    Jul 14, 2020 at 9:18
  • $\begingroup$ @kennysong - an unbiased model with high variance will presumably have a training error $\endgroup$
    – Henry
    Jul 16, 2020 at 1:38

3 Answers 3

1
$\begingroup$

I don't think so. Consider the case where we are trying to use a Gaussian Process to estimate an expensive, deterministic function (an emulator - not to be confused with the type of thing you run GBA games on). In essence to make an emulator, we run a computationally expensive computer model (e.g. a big physics simulation, a climate model or whatever) a handful of times and use the observed computer model runs to build a GP. Using the GP allows us to predict the output from our expensive model at just a fraction of the computational effort. They're pretty accurate too.

Within sample, the error will be exactly $0$ because we have constructed an interpolator. The emulator perfectly represents the simulator at the training points. Out of sample there will be some error. The GP emulator typically leverages the Bayesian framework so priors are placed over all hyperparameters of the GP. Therefore any point estimate derived from the posterior will exhibit some kind of bias. Although there are special cases of the GP prior which exhibit no bias!

$\endgroup$
5
  • $\begingroup$ I'm not familiar with GP, but I'll take your word that a biased GP model can have zero training error. What about the other direction? Does a model with no bias necessarily have zero training error? $\endgroup$
    – kennysong
    Jul 15, 2020 at 1:41
  • $\begingroup$ I don't think it would go the other way either. Consider a linear regression model, it's well known that the estimates (the $\beta$s and the $y$) are unbiased, but, because the data are noisy there will certainly be error in 'predicting' the observed values $\endgroup$
    – jcken
    Jul 15, 2020 at 7:44
  • $\begingroup$ Hm, I think we're using different definitions of bias. The canonical example of high bias for me is using linear regression when the data is not linear. For example, "models with higher bias tend to be relatively simple (low-order or even linear regression polynomials)" from here. $\endgroup$
    – kennysong
    Jul 15, 2020 at 13:32
  • $\begingroup$ Okay so even if your data are actually linear and you fit a linear regression to it, there will still be error in your predictions. The estimates however would be unbiased. $\endgroup$
    – jcken
    Jul 15, 2020 at 18:57
  • $\begingroup$ Ah, fair enough. That answers my original question (which was poorly worded), so accepted your answer. I was implicitly thinking of classification and 0/1 error – if a model is unbiased, does it classify all training points with perfect accuracy? $\endgroup$
    – kennysong
    Jul 15, 2020 at 23:24
3
$\begingroup$

The accepted answer is a bit complicated, so here's a simpler answer:

zero training error =/> zero bias

You can get zero training error by chance, with any model. Say your biased classifier always predicts zero, but your dataset happens to be all labeled zero.

zero bias =/> zero training error

Zero training error is impossible in general, because of Bayes error (think: two points in your training data are identical except for the label).

A better question would be, does zero bias imply that training error = Bayes error? I don't know.

$\endgroup$
1
$\begingroup$

The classical view of machine learning is that there is a bias-variance tradeoff and such a corresponding "sweet spot" between under-fitting and over-fitting. However, modern deep architectures can continue improving in test performance even after reaching very low or zero training risk and entering the "over-parameterized regime".

enter image description here

This is explored in the paper Reconciling modern machine learning practice and the bias-variance trade-off by Belkin-Hsu-Ma-Mandal 2018 (https://arxiv.org/abs/1812.11118).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.