What is Bayes Error in machine learning?

Question

http://www.deeplearningbook.org/contents/ml.html Page 116 explains bayes error as below

The ideal model is an oracle that simply knows the true probability distribution that generates the data. Even such a model will still incur some error on many problems, because there may still be some noise in the distribution. In the case of supervised learning, the mapping from x to y may be inherently stochastic, or y may be a deterministic function that involves other variables besides those included in x. The error incurred by an oracle making predictions from the true distribution p(x, y) is called the Bayes error.

Questions

1. Please explain Bayes error intuitively?
2. How is it different from irreducible error?
3. Can I say total error = Bias + Variance +Bayes error?
4. What is meaning of "y may be inherently stochastic"?

Answer 1

Bayes error is the lowest possible prediction error that can be achieved and is the same as irreducible error. If one would know exactly what process generates the data, then errors will still be made if the process is random. This is also what is meant by "$y$ is inherently stochastic".

For example, when flipping a fair coin, we know exactly what process generates the outcome (a binomial distribution). However, if we were to predict the outcome of a series of coin flips, we would still make errors, because the process is inherently random (i.e. stochastic).

To answer your other question, you are correct in stating that the total error is the sum of (squared) bias, variance and irreducible error. See also this article for an easy to understand explanation of these three concepts.

Answer 2

the essential of the statistic is the lack of information: ex: to determine the output of flip coin, we have to know earth gravitation at the test point, coin curvature, wind speed, hand posture,... If it is determined, it will surely know the output of that experiment. But we can't determine it all. Or in the determining price of the house, we have to know the location, market, macroeconomic,.. not only the distance to the center and the size of the house. => Therefore, in ML, if we have the training set only include the distance to the center and the size of the house, the output is still stochastic, not determinable, -> also have the error, even with the oracle (in Deep Learning book: “y may be a deterministic function that involves other variables besides those included in x “)

Answer 3

From https://www.cs.helsinki.fi/u/jkivinen/opetus/iml/2013/Bayes.pdf. For classification task, bayes error is defined as :

$min_f=Cost(f)$

Bayes Classifier is defined as: $argmin_f=Cost(f)$

So total error=bayes error + how much your model is worse than bayes error$\not\equiv$Bias + Variance +Bayes error which may depend on your model and the inherent nature of "distribution noise"

What is meaning of "y may be inherently stochastic"? For example, $y=f(x)=sin(x)$. But what you collect as y is always polluted as $\tilde{y}=y+t $, where $t\sim N(0, \sigma^2)$ So you have no way to know real y, and the cost estimation you have is inherently polluted. Even Oracle gives you the right answer, you think they are wrong.