# Maximum likelihood solution in classification problem

I have a couple of questions regarding the maximum likelihood solution in a classification problem (with only two classes $C_{1}$ and $C_{2}$) Basically, I have the following likelihood function:

$$p({\bf{t}}|\pi, \mu_{1},\mu_{2},\Sigma)=\Pi_{n=1}^{N}[\pi N(x_{n}|\mu_{1},\Sigma)]^{t_{n}}[(1-\pi)N(x_{n}|\mu_{2},\Sigma)]^{1-t_{n}}$$

in which $\bf{t}$ is the target vector, $P(C_{1})=\pi$ and consequently, the probability $P(C_{2}) = 1-\pi$. $C_{1}$ and $C_{2}$ have mean $\mu_{1}$ and $\mu_{2}$, respectively and both share the same covariance matrix $\Sigma$. Finally, $t_{n}$ is equal to one if the input vector $x_{n}$ belongs to $C_{1}$ and $0$ otherwise.

The first question is: Why the likelihood function seems to be the product of the joint distributions of $P(x_{n}, C_{1})$ and $P(x_{n}, C_{2})$:

$$P(x_{n}, C_{1})=P(C_{1})P(x_{n}|C_{1}),\;\; P(x_{n}, C_{2})=P(C_{2})P(x_{n}|C_{2})$$

The second question deals with the calculation of the maximum likelihood for a covariance matrix $\Sigma$. In the book I'm following (Bishop, Pattern Recognition and Machine Learning), it describes the solution in the following way:

with

$$S = \frac{N_{1}}{N}S_{1}+\frac{N_{2}}{N}S_{2}$$

$$S_{1}= \frac{1}{N_{1}}\Sigma_{n\in C_{1}}(x_{n}-\mu_{1})(x_{n}-\mu_{1})^{T}$$

$$S_{2}= \frac{1}{N_{2}}\Sigma_{n\in C_{2}}(x_{n}-\mu_{2})(x_{n}-\mu_{2})^{T}$$

How can I prove that the second and fourth term in the equation above can be expressed as a trace $\text{Tr}\{\Sigma^{-1}S\}$. I know that the first and third term give $-\frac{N}{2}\ln|\Sigma|$.

Thanks a lot!

First question: that expression applies only to training, and is a convinient form which allows to derive an analytical expression. Your target vector, $\mathbf{t}$, is a vector whose components are all zero except for that component coding the class that the sample, $x_{n}$, that is, $t_{n}$. Hence, it is equivalent to the expression for your mixture model,
$$p(x|π,μ1,μ2,Σ)=π\mathcal{N}(x|μ_{1},Σ)+(1−π)\mathcal{N}(x|μ_{2},Σ)$$
for the training data (beware the $t_{n}$'s). After training, you evaluate your mixture model using the expression above.
As for the second question: it is a refactorization of your expression. Note that $Tr$ is just the trace of that matrix, i.e. the elements of the diagonal of the matrix. To see that I find it easier to write explicitly the expression for $Tr{\Sigma^{-1}S}$ and group the terms to get the expression on the left.
• Sorry for the late response! Thank you. I just thought using the joint distribution was very weird. In regards to the second question, what expression would you use for $\Sigma^{-1}$ when writing the trace explicitly? – Robert Smith Apr 16 '13 at 3:44
• You can write the elements as $\sigma_{ij}$. The idea is that you can calculate $\Sigma^{-1}S$ can be calculated in two steps: $\Sigma^{-1}S = \Sigma^{-1}\left( \sum_{n \in C_{1}}\left(x_{n}-\mu_{1}\right)\left(x_{n}-\mu_{1}\right)^{T} \right) = \sum_{n \in C_{1}}\left(\Sigma^{-1}\left(x_{n}-\mu_{1}\right) \right) \left(x_{n}-\mu_{1}\right)^{T}$ – jpmuc Apr 17 '13 at 12:07