In machine learning, high bias prevents a model from properly fitting even the training set.
So, does a model have zero training error if and only if it has zero bias?
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.Sign up to join this community
I don't think so. Consider the case where we are trying to use a Gaussian Process to estimate an expensive, deterministic function (an emulator - not to be confused with the type of thing you run GBA games on). In essence to make an emulator, we run a computationally expensive computer model (e.g. a big physics simulation, a climate model or whatever) a handful of times and use the observed computer model runs to build a GP. Using the GP allows us to predict the output from our expensive model at just a fraction of the computational effort. They're pretty accurate too.
Within sample, the error will be exactly $0$ because we have constructed an interpolator. The emulator perfectly represents the simulator at the training points. Out of sample there will be some error. The GP emulator typically leverages the Bayesian framework so priors are placed over all hyperparameters of the GP. Therefore any point estimate derived from the posterior will exhibit some kind of bias. Although there are special cases of the GP prior which exhibit no bias!
The accepted answer is a bit complicated, so here's a simpler answer:
zero training error =/> zero bias
You can get zero training error by chance, with any model. Say your biased classifier always predicts zero, but your dataset happens to be all labeled zero.
zero bias =/> zero training error
Zero training error is impossible in general, because of Bayes error (think: two points in your training data are identical except for the label).
A better question would be, does zero bias imply that training error = Bayes error? I don't know.
The classical view of machine learning is that there is a bias-variance tradeoff and such a corresponding "sweet spot" between under-fitting and over-fitting. However, modern deep architectures can continue improving in test performance even after reaching very low or zero training risk and entering the "over-parameterized regime".
This is explored in the paper Reconciling modern machine learning practice and the bias-variance trade-off by Belkin-Hsu-Ma-Mandal 2018 (https://arxiv.org/abs/1812.11118).