In machine learning, high bias prevents a model from properly fitting even the training set.
So, does a model have zero training error if and only if it has zero bias?
$\begingroup$
$\endgroup$
6
3 Answers
$\begingroup$
$\endgroup$
5
I don't think so. Consider the case where we are trying to use a Gaussian Process to estimate an expensive, deterministic function (an emulator - not to be confused with the type of thing you run GBA games on). In essence to make an emulator, we run a computationally expensive computer model (e.g. a big physics simulation, a climate model or whatever) a handful of times and use the observed computer model runs to build a GP. Using the GP allows us to predict the output from our expensive model at just a fraction of the computational effort. They're pretty accurate too.
Within sample, the error will be exactly $0$ because we have constructed an interpolator. The emulator perfectly represents the simulator at the training points. Out of sample there will be some error. The GP emulator typically leverages the Bayesian framework so priors are placed over all hyperparameters of the GP. Therefore any point estimate derived from the posterior will exhibit some kind of bias. Although there are special cases of the GP prior which exhibit no bias!
-
$\begingroup$ I'm not familiar with GP, but I'll take your word that a biased GP model can have zero training error. What about the other direction? Does a model with no bias necessarily have zero training error? $\endgroup$ Jul 15, 2020 at 1:41
-
$\begingroup$ I don't think it would go the other way either. Consider a linear regression model, it's well known that the estimates (the $\beta$s and the $y$) are unbiased, but, because the data are noisy there will certainly be error in 'predicting' the observed values $\endgroup$– jckenJul 15, 2020 at 7:44
-
$\begingroup$ Hm, I think we're using different definitions of bias. The canonical example of high bias for me is using linear regression when the data is not linear. For example, "models with higher bias tend to be relatively simple (low-order or even linear regression polynomials)" from here. $\endgroup$ Jul 15, 2020 at 13:32
-
$\begingroup$ Okay so even if your data are actually linear and you fit a linear regression to it, there will still be error in your predictions. The estimates however would be unbiased. $\endgroup$– jckenJul 15, 2020 at 18:57
-
$\begingroup$ Ah, fair enough. That answers my original question (which was poorly worded), so accepted your answer. I was implicitly thinking of classification and 0/1 error – if a model is unbiased, does it classify all training points with perfect accuracy? $\endgroup$ Jul 15, 2020 at 23:24
$\begingroup$
$\endgroup$
The accepted answer is a bit complicated, so here's a simpler answer:
zero training error =/> zero bias
You can get zero training error by chance, with any model. Say your biased classifier always predicts zero, but your dataset happens to be all labeled zero.
zero bias =/> zero training error
Zero training error is impossible in general, because of Bayes error (think: two points in your training data are identical except for the label).
A better question would be, does zero bias imply that training error = Bayes error? I don't know.
$\begingroup$
$\endgroup$
The classical view of machine learning is that there is a bias-variance tradeoff and such a corresponding "sweet spot" between under-fitting and over-fitting. However, modern deep architectures can continue improving in test performance even after reaching very low or zero training risk and entering the "over-parameterized regime".
This is explored in the paper Reconciling modern machine learning practice and the bias-variance trade-off by Belkin-Hsu-Ma-Mandal 2018 (https://arxiv.org/abs/1812.11118).
function(x) return(0)and you are using it with train data that has only0values as labels, so your model is clearly biased (makes predictions independent of data), but the train error would be 0. $\endgroup$