1
$\begingroup$

I have a couple of questions regarding the maximum likelihood solution in a classification problem (with only two classes $C_{1}$ and $C_{2}$) Basically, I have the following likelihood function:

$$p({\bf{t}}|\pi, \mu_{1},\mu_{2},\Sigma)=\Pi_{n=1}^{N}[\pi N(x_{n}|\mu_{1},\Sigma)]^{t_{n}}[(1-\pi)N(x_{n}|\mu_{2},\Sigma)]^{1-t_{n}}$$

in which $\bf{t}$ is the target vector, $P(C_{1})=\pi$ and consequently, the probability $P(C_{2}) = 1-\pi$. $C_{1}$ and $C_{2}$ have mean $\mu_{1}$ and $\mu_{2}$, respectively and both share the same covariance matrix $\Sigma$. Finally, $t_{n}$ is equal to one if the input vector $x_{n}$ belongs to $C_{1}$ and $0$ otherwise.

The first question is: Why the likelihood function seems to be the product of the joint distributions of $P(x_{n}, C_{1})$ and $P(x_{n}, C_{2})$:

$$P(x_{n}, C_{1})=P(C_{1})P(x_{n}|C_{1}),\;\; P(x_{n}, C_{2})=P(C_{2})P(x_{n}|C_{2})$$

The second question deals with the calculation of the maximum likelihood for a covariance matrix $\Sigma$. In the book I'm following (Bishop, Pattern Recognition and Machine Learning), it describes the solution in the following way:

enter image description here

with

$$S = \frac{N_{1}}{N}S_{1}+\frac{N_{2}}{N}S_{2}$$

$$S_{1}= \frac{1}{N_{1}}\Sigma_{n\in C_{1}}(x_{n}-\mu_{1})(x_{n}-\mu_{1})^{T}$$

$$S_{2}= \frac{1}{N_{2}}\Sigma_{n\in C_{2}}(x_{n}-\mu_{2})(x_{n}-\mu_{2})^{T}$$

How can I prove that the second and fourth term in the equation above can be expressed as a trace $\text{Tr}\{\Sigma^{-1}S\}$. I know that the first and third term give $-\frac{N}{2}\ln|\Sigma|$.

Thanks a lot!

$\endgroup$
2
$\begingroup$

First question: that expression applies only to training, and is a convinient form which allows to derive an analytical expression. Your target vector, $\mathbf{t}$, is a vector whose components are all zero except for that component coding the class that the sample, $x_{n}$, that is, $t_{n}$. Hence, it is equivalent to the expression for your mixture model,

$$p(x|π,μ1,μ2,Σ)=π\mathcal{N}(x|μ_{1},Σ)+(1−π)\mathcal{N}(x|μ_{2},Σ)$$

for the training data (beware the $t_{n}$'s). After training, you evaluate your mixture model using the expression above.

As for the second question: it is a refactorization of your expression. Note that $Tr$ is just the trace of that matrix, i.e. the elements of the diagonal of the matrix. To see that I find it easier to write explicitly the expression for $Tr{\Sigma^{-1}S}$ and group the terms to get the expression on the left.

$\endgroup$
  • $\begingroup$ Sorry for the late response! Thank you. I just thought using the joint distribution was very weird. In regards to the second question, what expression would you use for $\Sigma^{-1}$ when writing the trace explicitly? $\endgroup$ – Robert Smith Apr 16 '13 at 3:44
  • $\begingroup$ You can write the elements as $\sigma_{ij}$. The idea is that you can calculate $\Sigma^{-1}S$ can be calculated in two steps: $\Sigma^{-1}S = \Sigma^{-1}\left( \sum_{n \in C_{1}}\left(x_{n}-\mu_{1}\right)\left(x_{n}-\mu_{1}\right)^{T} \right) = \sum_{n \in C_{1}}\left(\Sigma^{-1}\left(x_{n}-\mu_{1}\right) \right) \left(x_{n}-\mu_{1}\right)^{T}$ $\endgroup$ – jpmuc Apr 17 '13 at 12:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.