Linear discriminant analysis likelihood function

Question

It's a lot of time that I don't do probability stuff, so probably this is a very naive question for most of you. Anyway..

For the linear (fisher) discriminant we have an inpu $X$ and an output $Y$ and, in particular, we assume that

$$ Y\sim Bernoulli(\pi)$$ $$ X|Y=k \sim N(x|\mu_k,\Sigma)$$

Then, when I want to obtain the likelihood function, I do

$$log~p(x,y) = \sum_{i=1}^N logp(x_i,y_i)\\= \sum_{i=1}^N log\{[\pi N(x_i|\mu_1,\Sigma)]^{y_i} [(1-\pi) N(x_i|\mu_1,\Sigma)]^{1-y_i} \}$$

but I don't undesrtand why also $N(x_i|\mu,\Sigma)$ is to the power of $y_i$ and $ (1-y_i)$ or, in other words, why it's not like this:

$$log ~p(x,y) = \sum_{i=1}^N logp(x_i,y_i)\\= \sum_{i=1}^N log\{[\pi^{y_i} N(x_i|\mu_1,\Sigma)] [(1-\pi)^{1-y_i} N(x_i|\mu_1,\Sigma)] \}$$

Since all I'm doing is multiplying (for the chain rule) $p(y) = \pi^{y}(1-\pi)^{1-y}$ and $p(x|y=k) = N(x|\mu_k,\Sigma) = \frac{1}{(2\pi|\Sigma|)^{\frac{1}{2}}} \exp\{-\frac{1}{2}(x - \mu_k)^T\Sigma^{-1}(x-\mu_k)\}$

Answer 1

You need to use the rule for conditional probability, $p(x_i,y_i) = p(x_i|y_i)p(y_i)$, for each of the two cases $y_i=0$ and $y_i=1$.

When $y_i=1$, the probability density $p(x_i,y_i)$ is \begin{split} p(x_i,y_i=1) & = \color{red}{p(y_i=1)} \color{blue}{p(x_i|y_i=1)} \\ & = \color{ red}{\pi} \color{blue}{N(x_i|\mu_1,\Sigma)} \end{split}

Similarly when $y_i=0$, the probability density $p(x_i,y_i)$ is $(1-\pi) N(x_i|\mu_0,\Sigma)$.

When you merge the two cases together, the density becomes $[(1-\pi) N(x_i|\mu_0,\Sigma)]^{1-y_i}[\pi N(x_i|\mu_1,\Sigma)]^{y_i}$. Try plugging in $y_i = 0$ and $y_i=1$ and see whether you can recover the results above.

In neither of the cases, is the density proportional to $N(x_i|\mu_1,\Sigma)N(x_i|\mu_0,\Sigma)$, which is what you have when the power of $y_i$ is not shared by $N(x_i|\mu_k,\Sigma)$. More specifically, what you propose is this

\begin{equation} p(x_i,y_i) = \{ \begin{array}{lc} \pi N(x_i|\mu_1,\Sigma)N(x_i|\mu_0,\Sigma) & \text{when } y_i=1 \\ (1-\pi) N(x_i|\mu_1,\Sigma)N(x_i|\mu_0,\Sigma) & \text{when } y_i=0 \end{array} \end{equation}

In other words, you have to turn on\off $N(x_i|\mu_k,\Sigma)$ at the same time as $p(y_i)$.