Bayes decision theory: Classification error probability

Question

In Bayesian decision theory: Given $\omega_1$ and $\omega_2$ as two classes for classification, $P\left( \omega_1 \right)$ and $P\left( \omega_2\right)$ their prior probabilities, $x$ the feature vector representing an unknown pattern, $P\left( \omega_1 | x\right)$ and $P\left( \omega_2 | x\right)$ as posteriori probabilities, $p\left( x |\omega_i\right)$ the likelihood function of $\omega_i$ with respect to $x$; and $R_i$ the region of feature space where decision is in favor of $\omega_i$

why is it that the minimization of the probability error $P_e$ , given by:

\begin{align*} P_e &= P\left( x \in R_1, \omega_2 \right) + P\left( x \in R_2, \omega_1 \right)\\ &=P\left(\omega_2 \right) \int \limits_{R_1}p\left( x |\omega_2 \right) dx + P\left(\omega_1 \right) \int \limits_{R_2}p\left( x |\omega_1 \right)dx\\ &= \int \limits_{R_1}P\left( \omega_2 | x\right)p(x) dx + \int \limits_{R_2}P\left(\omega_1 | x \right)p(x)dx \end{align*}

by choosing regions $R_1$ and $R_2$ of feature space so that:

\begin{align} R_1 &: P\left(\omega_1 | x \right) > P\left( \omega_2 | x\right)\\ R_2 &: P\left(\omega_2 | x \right) > P\left( \omega_1 | x\right) \end{align}

is said to be not always the best for minimizing $P_e$ ?

Answer 1

Now I found that assertion. Actually, that is not exactly what the author of the book claims. First, I believe that you understand where that Bayes decision rule comes from. It is demonstrated in that book.

So the problem with this approach is, that all sort of mistakes are weighted equally, that is, confusing class $\omega_{2}$ for $\omega_{1}$ or viceversa is treated equally. There is no fundamental difference. However, in practice this is not the case. It is not the same saying that you don't suffer an illness when you actually have it, than otherwise. And so on. So there are cases where you need to weight them differently.

When using the Bayes decision criterion you find the minimum classification error, i.e. the maximum accuracy. But for diagnosis for example, you want to minimize the false negative rate (it is better to think that you need to make further tests that to release a sick person).

And that is why the author says it is not always optimal to use that decision rule, and goes on in the book introducing a weighted error function. So, basically, you do not minimize the same $P_{e}$.

Answer 2

Here's an attempt, although I don't feel that confident about it. Perhaps someone can comment on what I'm missing? Here goes: $$ \begin{align*} P_e &= P\left( x \in R_1, \omega_2 \right) + P\left( x \in R_2, \omega_1 \right) \\ &=P\left(\omega_2 \right) P(x \in R_1 \mid \omega_2) + P\left(\omega_1 \right) P(x \in R_2 \mid \omega_1) \\ \end{align*} $$ $P(\omega_2)$ and $P(\omega_1)$ are fixed, so let's try and choose $R_1, R_2$ such that $P(x \in R_2 \mid \omega_1)$ and $P(x \in R_1 \mid \omega_2)$ are minimized.

We do this through considering the integral $$ P(x \in R_2 \mid \omega_1) = \int_{R_2} p(x \mid \omega_1)dx. $$

Naturally, for each point $x \in R_1 \cup R_2$, we have that either $$ A: p(x \mid \omega_2) \ge p(x \mid \omega_1), $$ or that $$ B: p(x \mid \omega_2) < p(x \mid \omega_1). $$

Now let's construct $R_1, R_2$. First assume that the entire feature space is $R_1$ and $R_2 = \emptyset$. For each point $y \in R_1$ consider whether $A$ or $B$ holds. If $A$, then let $R_2 := R_2 \cup y$ and $R_1 := R_1 - {y}$. If $B$, do nothing.

When this is done, it is clear that you could take no points out of $R_1$ and append them to $R_2$ to reduce $P_e$, and vice versa. Thus we have minimized $P_e$ with respect to $R_1, R_2$. Further, we have their definition: $$ R_1 = \{x \mid p(x \mid \omega_2) < p(x \mid \omega_1)\}\\ R_2 = \{x \mid p(x \mid \omega_2) \ge p(x \mid \omega_1)\}. $$ This is distinct from choosing the regions that compare class probabilities given each point, as $p(x \mid \omega_2) = P(\omega_2 \mid x)$ is not true in general. It is perhaps worth noting that, with respect to minimizing $P_e$, $R_1$ and $R_2$ are not unique.. one could change which region gets the greater than symbol $\ge$ in the definition. Further, the integral does not take into account 'discontiguousities' in the region.