1
$\begingroup$

Very close to: Joint Gaussian of two Gaussians

I am trying to find the Bayesian classifier for two classes given by the following bivariate Gaussian distributions:

$$p(x|c_1) = N(\mu_1, \Sigma_1)$$ $$p(x|c_2) = N(\mu_2, \Sigma_2)$$

What is the formula and the methodology I should use to do this? I am confused with out a proper understanding on a formula to do that at the moment. Prior probabilities of $c_1$ and $c_2$ are 0.5 respectively.

FYI:

μ1 = ( 2        Σ1 = (2 0            
       2 )            0 1)


μ2 = ( 2        Σ1 = (4 0            
       4 )            0 2)   
$\endgroup$
  • $\begingroup$ Please rather use edits and comments than new question if you only want to ask for extending the answer. $\endgroup$ – user88 Mar 18 '12 at 17:04
3
$\begingroup$

Define a random variable $C\in\{1,2\}$ with prior distribution $\mu_C$ given by $$ \mu_C(A) = P\{C\in A\} = \frac{1}{2} I_A(1) + \frac{1}{2} I_A(2) \, , $$ where $A$ is any subset of $\{1,2\}$.

Use the notation $X=(X_1,X_2)$ and $x=(x_1,x_2)$. Suppose that

$$X\mid C=1\sim N(\mu_1,\Sigma_1)\, ,$$ $$X\mid C=2\sim N(\mu_2,\Sigma_2)\, ,$$

where $\mu_1=(2, 2)^\top$, $\Sigma_1=\textrm{diag}(2,1)$, $\mu_2=(2,4)^\top$ and $\Sigma_2=\textrm{diag}(4,2)$.

Now, study this

http://en.wikipedia.org/wiki/Multivariate_normal_distribution

to understand that

$$ f_{X\mid C}(x\mid 1) = \frac{1}{2\pi\sqrt{2}} \exp\left(-\frac{1}{2}\left(\frac{(x_1-2)^2}{2} + \frac{(x_2-2)^2}{1} \right)\right) \, , $$

$$ f_{X\mid C}(x\mid 2) = \frac{1}{4\pi\sqrt{2}} \exp\left(-\frac{1}{2}\left(\frac{(x_1-2)^2}{4} + \frac{(x_2-4)^2}{2} \right)\right) \, . $$

Using Bayes Theorem, we have $$ P\{C=1\mid X=x\} = \frac{\int_{\{1\}} f_{X\mid C}(x\mid c) \,d\mu_C(c)}{\int_{\{1,2\}} f_{X\mid C}(x\mid c)\, d\mu_C(c)} = \frac{\frac{1}{2} f_{X\mid C}(x\mid 1)}{\frac{1}{2} f_{X\mid C}(x\mid 1) + \frac{1}{2} f_{X\mid C}(x\mid 2)} \, . $$

The idea is to decide for the first classification if $$ P\{C=1\mid X=x\} = \frac{1}{1+\frac{f_{X\mid C}(x\mid 2)}{f_{X\mid C}(x\mid 1)}} > \frac{1}{2} \, , $$ which is equivalent to $$ \frac{f_{X\mid C}(x\mid 2)}{f_{X\mid C}(x\mid 1)} < 1 \, , $$ or $$ \log f_{X\mid C}(x\mid 2) - \log f_{X\mid C}(x\mid 1) < 0 \, , $$ which gives us $$ \log \frac{1}{2} - \frac{(x_1-2)^2}{8} - \frac{(x_2-2)^2}{4} + \frac{(x_1-2)^2}{4} + \frac{(x_2-4)^2}{2} < 0 \, . \qquad (*) $$ Therefore, you decide that the point $x$ belongs to classification $1$ if it is inside the ellipse defined by $$ \frac{(x_1-2)^2}{8(2+\log 2)} + \frac{(x_2-6)^2}{4(2+\log 2)} = 1 \, , $$ otherwise, you decide for classification $2$.

$\endgroup$
2
$\begingroup$

$X_1$ and $X_2$ are independent normal random variables with means $2$ and $4$ under both hypotheses. The variances under one hypothesis are half those under the other. So, the log likelihood ratio is something like

$$\ln 2 - \frac{(x_1-2)^2}{8}-\frac{(x_2-4)^2}{4} + \frac{(x_1-2)^2}{4}+\frac{(x_2-4)^2}{2}$$

and has to be compared to $0$ since the hypotheses are equally likely. After simplification you should get a classifier saying something

Decide $c_1$ if $(x_1,x_2)$ lies inside an ellipse with center $(2,4)$; $c_2$ if outside.

where the equation of the ellipse wil be revealed to you as you do the algebra.

$\endgroup$
  • $\begingroup$ Thank you very much for your answer Dilip. Still there are certain things that I don't understand maybe because I am pretty new to stat stuff. What does happen to the vector properties of the means (μ) and where are the co variance matrices represented in the equation you are giving? What you are giving is after substituting the values right? Do you have the bare bone equation which I can use in the classification? Thanks. $\endgroup$ – picmate Mar 16 '12 at 23:36
  • $\begingroup$ See, for example, here for the bivariate normal density formula. The barebones equation for the likelihood ratio is the ratio of the two bivariate densities under the two hypotheses. The covariances are zero according to the FYI statement and so $X_1$ and $X_2$ are independent normal random variables while the means are there subtracting off from x_1 and x_2 in the formula I gave. I took the log of the likelihood ratio to get the formula. $\endgroup$ – Dilip Sarwate Mar 17 '12 at 1:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.