When we train a classifier to predict $y \in \{1, \dots, K\}$ given an input $x$, classification is done by reporting the class with the highest posterior probability as the prediction; that is: $$ \hat{y}(x) = \arg\max_{\hspace{-0.75cm} y \in \{1, \dots, K\}} P(y | x) $$ The justification for such a prediction rule, as far as I know, it that this gives the Bayes optimal classifier.

This makes sense to me only if $P(y | x)$ matches the true distribution of the data exactly. However, $P(y | x)$ is only an estimated model that we fit to the training data, probably after making lots of incorrect assumptions, and most likely does not perfectly match the distribution of the actual data.

My question is, does the Bayes optimal classification rule still applies to estimated models, or does it only hold for the true underlying distribution of the data.


1 Answer 1


Why classifiers report the class with maximum posterior probability as the predicted class?

Because if we knew the true model, we wouldn't be training a classifier :)

If you are instead referring to the Bayes Optimal Classifer, keep in mind this is a theoretical construct for reasoning about the discriminative power (VC dimension) that a certain class of hypothesis may hold and is, of course, not a realistic classifier to train.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.