# Questions regarding the Bayes Classifier in *Introduction to Statistical Learning*

I am having trouble grokking some very elementary material regarding Bayesian Classification in Introduction to Statistical Learning at the end of pg. 37 to the very top of pg. 39 (i.e., the section entitled "The Bayes Classifier" which is accessible via the link).

Here is a relevant snippet:

It is possible to show (though the proof is outside of the scope of this book) that the test error rate given in (2.9) is minimized, on average, by a very simple classifier that assigns each observation to the most likely class, given its predictor values. In other words, we should simply assign a test observation with predictor vector $x_0$ to the class $j$ for which

$$Pr(Y= j \mid X = x_0)$$

is largest.

1. Where does the probability distribution come from to begin with? Is it inferred from the training data? Or is what is inferred from the training data a mere approximation of the real probability distribution (wherever it comes from)?

2. (Note: for this question you need to see the figure on page 38 pdf linked to above.) Need the Bayes Decision boundary be a straight line splitting the predictors into neat, contiguous boundaries? The example used in this section seems awfully convenient. Why can't there be a hodge podge of probability neighborhoods that aren't even separated by a single, clean line? Is there something about the Bayes Classification boundary definition that prevents this?

• Isn't it true that there might not exist any Bayesian decision boundary at all? Suppose every other point in $X$ were to oscillate between categories. No such line(s) could be drawn. Moreover, every point in the space could have a 50% chance of being in either category (in which case the entire space would be a "boundary"). Thus there might not even be a set of bayesian decision boundary lines to begin with. Is this correct? – George Apr 22 '16 at 15:03
• @George Oh, now I understand your question. Let's take some point $x_0$ in feature space (on the figure feature space is 2-dimensional). We can calculate conditional probability $Pr(Y = j | X = x_0)$. On the figure there are two classes. And we predict that $x_0$'s class is $\arg \max\limits_{j \in Y} Pr(Y = j | X = x_0)$. – Maassa Apr 22 '16 at 16:20