I am having trouble grokking some very elementary material regarding Bayesian Classification in Introduction to Statistical Learning at the end of pg. 37 to the very top of pg. 39 (i.e., the section entitled "The Bayes Classifier" which is accessible via the link).
Here is a relevant snippet:
It is possible to show (though the proof is outside of the scope of this book) that the test error rate given in (2.9) is minimized, on average, by a very simple classifier that assigns each observation to the most likely class, given its predictor values. In other words, we should simply assign a test observation with predictor vector $x_0$ to the class $j$ for which
$$ Pr(Y= j \mid X = x_0) $$
Where does the probability distribution come from to begin with? Is it inferred from the training data? Or is what is inferred from the training data a mere approximation of the real probability distribution (wherever it comes from)?
(Note: for this question you need to see the figure on page 38 pdf linked to above.) Need the Bayes Decision boundary be a straight line splitting the predictors into neat, contiguous boundaries? The example used in this section seems awfully convenient. Why can't there be a hodge podge of probability neighborhoods that aren't even separated by a single, clean line? Is there something about the Bayes Classification boundary definition that prevents this?