# Bayes classifier

Given the prior probability of 2 distributions, $N(x,y)$ and $N(a,b)$, where $N(\mu,\sigma^2)$:

How do you make a decision rule to minimize the probability of error, if the prior probabilities are equal? Can you give an example?

What if the prior probabilities are different, such as
$P(\text{Distribution 1}) = 0.70$
$P(\text{Distribution 2}) = 0.30$?

-
I am little bit confused. My first guess was that you are trying to predict the probability of occurrence for a value of N(.,.), but your heading says "classifier". What is the label/target/response variable you are trying to predict ? –  steffen Mar 11 '11 at 21:18
I think he tries to do basically the example from chapter 1 of the Duda-Hart book. There are two classes, points from one are distributed normally with parameters x, y, points from another are normal with params a, b. What is the posterior probability of an observation given each class, and what is the decision rule. –  SheldonCooper Mar 11 '11 at 21:23
@SheldonCooper yes!. Do you have some example. or a link to solve this in MATLAB?? –  cMinor Mar 11 '11 at 22:32
Thank you Sheldon. @Question: (+1) after mbq's edit. –  steffen Mar 12 '11 at 15:23

Let's say we have to classes $C$, i.e. $c_1$ and $c_2$ and further that the points of the class $c_1$ are drawn from N(a,b) and the points of $c_2$ from N(c,d) respectively (note the changes in notation). The point variable shall be X, a value of this variable x.

Here is the Bayes-Theorem: $p(C|X)=\frac{p(X|C)*p(C)}{p(X)}$

The bayes decision rule for binary classification states: Given x, one decides...

• in favor of $c_1$ If $p(c_1|x) > p(c_2|x)$
• in favor of $c_2$ If $p(c_2|x) > p(c_1|x)$

The cases that both posterior probabilities is equal has to be treated separately. Normally one favors one class more than the other and hence decides in favor of that class (if a decision is forced). Let's say, that if the posteriors are equal, we decide in favor of class $c_2$

The above decision rule can be reformulated to: If $\frac{p(c_1|x)}{p(c_2|x)} > 0$, decide in favor of $c_1$, else $c_2$

Applying Bayes Theorem ... $\frac{p(c_1|x)}{p(c_2|x)}$ = $\frac{p(x|c_1)*p(c_1)}{p(x|c_2)*p(c_2)}$

If the priors are equal, the decision rule resolves to $\frac{p(x|c_1)}{p(x|c_2)}$.

If p(c_1)=0.7 and p(c_2)=0.3, the decision rule resolves to $\frac{0.7}{0.3}*\frac{p(x|c_1)}{p(x|c_2)}\approx 2.33*\frac{p(x|c_1)}{p(x|c_2)}$

The conditional densities p(X|C) are defined by the corresponding normal distributions. The density of a normal distribution for (not at) a point x can be calculated as $pdf(x|N(a,b))=\frac{1}{\sqrt{2\pi*b}} * e^{-\frac{(x-a)^2}{2b}}$.

Inserting this into our formula (for case of equal priors) one gets $\frac{p(x|c_1)}{p(x|c_2)}=\frac{pdf(x|N(a,b))}{pdf(x|N(c,d))}$. Starting from here you can "simplify" the formula even further.

Edit: Here is some R - code to visualize the priors * conditional probabilities (ignoring the normalization factor p(X)):

require(lattice)
#
set.seed(42)

# parameters for distribution of class 1
a <- 0
b <- 1
# parameters for distribution of class 2
c <- 3
d <- 1

x <- c(sort(rnorm(1000,mean=a,sd=b)),sort(c(rnorm(1000,mean=c,sd=d))))
y <- c(dnorm(x[1:1000],mean=a,sd=b),dnorm(x[1001:2000],mean=c,sd=d))
labels <- factor(rep(c("class 1","class 2"),each=1000))

dat <- data.frame("x"=x,"density"=y,"groups"=labels)
xyplot(density~x,data=dat,groups=labels,type="b",auto.key=T)


which results in this plot, where you can see the decision "rule" (line). Starting from here, I suggest to play around a little bit with the priors to get a good feeling "what's going on".

-