I am having trouble grokking some very elementary material regarding Bayesian Classification in Introduction to Statistical Learning at the end of pg. 37 to the very top of pg. 39 (i.e., the section entitled "The Bayes Classifier" which is accessible via the link).

Here is a relevant snippet:

It is possible to show (though the proof is outside of the scope of this book) that the test error rate given in (2.9) is minimized, on average, by a very simple classifier that assigns each observation to the most likely class, given its predictor values. In other words, we should simply assign a test observation with predictor vector $x_0$ to the class $j$ for which

$$ Pr(Y= j \mid X = x_0) $$

is largest.

  1. Where does the probability distribution come from to begin with? Is it inferred from the training data? Or is what is inferred from the training data a mere approximation of the real probability distribution (wherever it comes from)?

  2. (Note: for this question you need to see the figure on page 38 pdf linked to above.) Need the Bayes Decision boundary be a straight line splitting the predictors into neat, contiguous boundaries? The example used in this section seems awfully convenient. Why can't there be a hodge podge of probability neighborhoods that aren't even separated by a single, clean line? Is there something about the Bayes Classification boundary definition that prevents this?


1 Answer 1

  1. We make an assumption that all objects are realizations of random variables, which are independent and identically distributed from some distribution. Very often we have one more assumption that this distribution is from exact parametric family of distributions. In this case training data is used to evaluate its parameters.

  2. No, in general case the Bayes decision boundary shouldn't be linear. If you want linear decision boundary, look for linear classifiers such as SVM, logistic regression and the Naive Bayes classifier.

  • $\begingroup$ Isn't it true that there might not exist any Bayesian decision boundary at all? Suppose every other point in $X$ were to oscillate between categories. No such line(s) could be drawn. Moreover, every point in the space could have a 50% chance of being in either category (in which case the entire space would be a "boundary"). Thus there might not even be a set of bayesian decision boundary lines to begin with. Is this correct? $\endgroup$
    – George
    Apr 22, 2016 at 15:03
  • $\begingroup$ @George Oh, now I understand your question. Let's take some point $x_0$ in feature space (on the figure feature space is 2-dimensional). We can calculate conditional probability $Pr(Y = j | X = x_0)$. On the figure there are two classes. And we predict that $x_0$'s class is $\arg \max\limits_{j \in Y} Pr(Y = j | X = x_0)$. $\endgroup$
    – Maassa
    Apr 22, 2016 at 16:20
  • $\begingroup$ @George All points of feature space will be separated into parts. And decision boundary can be viewed as a line, which separates them. $\endgroup$
    – Maassa
    Apr 22, 2016 at 16:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.