I am new to Bayesian Decision Theory and don't understand the following concept:

So from what I understood, the Bayes error is used to report the performance of a Bayes classifier in terms of the probability of making and error. From the conditional error probabilities conditional error http://sebastianraschka.com/_my_resources/images/equations/cond_error.png

we can obtain the total probability of making an error (the probability to mis-classify).

error http://sebastianraschka.com/_my_resources/images/equations/error.png

Now, if my Bayes classifier was designed to minimize the overall risk, I have a loss function that gives penalties to certain decisions.

conditional risk http://sebastianraschka.com/_my_resources/images/equations/cond_risk.png overall risk http://sebastianraschka.com/_my_resources/images/equations/overall_risk.png

So, if my classifier includes such a loss function when I optimize my classifier for minimum overall risk, shouldn't be the Bayes error also include the loss function term?

Hope you can help me here, because I think I am missing something here ...


I'll try to express my problem using an 2D-classification problem:

Let's assume I have two pdfs (e.g., p(x|c1) and p(x|c2) ) with slight overlap. And mis-classifying a pattern as c2 where it truly belongs to c1 is more costly than vice-versa.

In this case I would assign a higher loss to "classify pattern x as c1 when it is truly c2" than "classify pattern x as c2 when it is truly c1" in order to calculate and minimize the overall risk.

I would therefore increase the probability to classify a pattern x as c2 over c1 due to the minimum risk optimization. Isn't this something I have to also include in p(error)?


1 Answer 1


You should be incorporating it, yes. The loss function will affect your policy on how to classify a pattern. Let's call that policy $\pi$, and say it consists of the probability you assign to each class. You thus have a set of probabilities $p(\textrm{choose } c_i|x,\pi) $, the probability of assigning pattern $x$ to class $c_i$ under policy $\pi$.

$P(\textrm{error}|x)$, as you've written it, has separate cases depending on your choice of classification: for example, your first equation has two cases, depending on whether you choose class $\omega_1$ or $\omega_2$.

However, instead of $P(\textrm{error} )$, what you really want to know is $P(\textrm{error}|\pi)$, i.e. the chance of an error given your policy on classification. So you also need to sum over all of these possible cases. For your example, you'd have $$\begin{align*}P(\textrm{error}|\pi) &= \int_{-\infty}^{\infty} P(\textrm{error} |x,\pi) p(x) dx \\&= \int_{-\infty}^{\infty} \sum_{i=1}^2 P(\textrm{error} |x,\textrm{choose } c_i) p(\textrm{choose } c_i|x,\pi) p(x) dx.\end{align*} $$ You could also write $P(\textrm{error} |R) ,$ and condition on the loss function $R$ directly, but your choice of $\pi$ given $R$ is usually deterministic, so it won't make a difference.

  • $\begingroup$ Thanks a lot. Meanwhile I did some further reading, and I guess I was a little bit confused about the term "Bayes Error". When I understand correctly, P(errror|π) would give me the error given my decision policy (which includes the loss function), and P(error) is just a theoretical scenario that describes the best possible classification accuracy we can achieve. And I assume if we have a 1-0 loss function, P(error|π) would be equal to P(error)? $\endgroup$
    – user39663
    Feb 10, 2014 at 6:06
  • 1
    $\begingroup$ I think so, yes. If you have a 1-0 loss function, then the loss is minimised by choosing the most likely class every time. In other words, you're minimising P(error|x) for all x, which just gives you P(error). In other words, if your loss function is chosen as the chance of you being wrong, it minimises the chance of you being wrong! $\endgroup$ Feb 10, 2014 at 11:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy