3
$\begingroup$

According to the Wikipedia, the Bayes classifier assumes knowledge of the distributions of $X | Y$, where $X$ and $Y$ are the random variables of the features and the classes, respectively. Let's assume that one knows that these are normally distributed and decides to estimate the parameters of the normal distribution from the training sample via MLE.

Would the classifier

$$ \hat c = \underset{c}{\operatorname{arg\,max}} \; Pr[X|Y=c]\,Pr[Y=c] $$

based on the estimated distributions, still be considered a Bayes classifier, or is the Bayes classifier a theoretical construct only defined on the true data distributions?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
0
$\begingroup$

Optimal Bayes classifier is the classifier where you make the classification decision based on the known conditional distribution

$$ \hat c = \underset{c}{\operatorname{arg\,max}} \; P(Y=c|X) $$

Knowing the distribution makes it optimal. Most, if not all, machine learning models can be defined in probabilistic terms as models that try to approximate the conditional distribution. We do not call those models Bayes classifiers, just classifiers. The name is reserved for the theoretical construct.

Improve this answer
$\endgroup$
4
  • $\begingroup$ I see. would the error of such "MLE-Bayes" classifier be considered a good estimation of the Bayes error? $\endgroup$
    – synack
    Commented Jul 31, 2023 at 14:01
  • $\begingroup$ @synack could you give an example? $\endgroup$
    – Tim
    Commented Jul 31, 2023 at 15:07
  • $\begingroup$ following up on the earlier example, imagine a simple learning problem where there is only one feature $X\in \mathbb R$, only two classes, $Pr[Y=0]=Pr[Y=1]$, and $X|Y=0\sim\mathcal N(-1, 1)$ and $X|Y=1\sim\mathcal N(1, 1)$. The decision boundary of the Bayes classifier would lie in $x=0$. However, let's assume we do not know the parameters of these normal distributions and we estimate them using the sample mean and the sample variance on a 'training' dataset and then use the decision rule of the Bayes classifier, would the error of this classifier be a good estimate of the Bayes error? $\endgroup$
    – synack
    Commented Aug 1, 2023 at 9:32
  • $\begingroup$ @synack we know that no estimator can have a better error than the Bayes optimal estimator. Something else than Bayes optimal estimator would have a higher error. If you take some estimator and use it as an approximation of the Bayes error, you cannot claim anymore that nothing can be better than this, unless you can prove it otherwise. $\endgroup$
    – Tim
    Commented Aug 1, 2023 at 9:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.