It's a lot of time that I don't do probability stuff, so probably this is a very naive question for most of you. Anyway..

For the linear (fisher) discriminant we have an inpu $X$ and an output $Y$ and, in particular, we assume that

$$ Y\sim Bernoulli(\pi)$$ $$ X|Y=k \sim N(x|\mu_k,\Sigma)$$

Then, when I want to obtain the likelihood function, I do

$$log~p(x,y) = \sum_{i=1}^N logp(x_i,y_i)\\= \sum_{i=1}^N log\{[\pi N(x_i|\mu_1,\Sigma)]^{y_i} [(1-\pi) N(x_i|\mu_1,\Sigma)]^{1-y_i} \}$$

but I don't undesrtand why also $N(x_i|\mu,\Sigma)$ is to the power of $y_i$ and $ (1-y_i)$ or, in other words, why it's not like this:

$$log ~p(x,y) = \sum_{i=1}^N logp(x_i,y_i)\\= \sum_{i=1}^N log\{[\pi^{y_i} N(x_i|\mu_1,\Sigma)] [(1-\pi)^{1-y_i} N(x_i|\mu_1,\Sigma)] \}$$

Since all I'm doing is multiplying (for the chain rule) $p(y) = \pi^{y}(1-\pi)^{1-y}$ and $p(x|y=k) = N(x|\mu_k,\Sigma) = \frac{1}{(2\pi|\Sigma|)^{\frac{1}{2}}} \exp\{-\frac{1}{2}(x - \mu_k)^T\Sigma^{-1}(x-\mu_k)\}$


1 Answer 1


You need to use the rule for conditional probability, $p(x_i,y_i) = p(x_i|y_i)p(y_i)$, for each of the two cases $y_i=0$ and $y_i=1$.

When $y_i=1$, the probability density $p(x_i,y_i)$ is \begin{split} p(x_i,y_i=1) & = \color{red}{p(y_i=1)} \color{blue}{p(x_i|y_i=1)} \\ & = \color{ red}{\pi} \color{blue}{N(x_i|\mu_1,\Sigma)} \end{split}

Similarly when $y_i=0$, the probability density $p(x_i,y_i)$ is $(1-\pi) N(x_i|\mu_0,\Sigma)$.

When you merge the two cases together, the density becomes $[(1-\pi) N(x_i|\mu_0,\Sigma)]^{1-y_i}[\pi N(x_i|\mu_1,\Sigma)]^{y_i}$. Try plugging in $y_i = 0$ and $y_i=1$ and see whether you can recover the results above.

In neither of the cases, is the density proportional to $N(x_i|\mu_1,\Sigma)N(x_i|\mu_0,\Sigma)$, which is what you have when the power of $y_i$ is not shared by $N(x_i|\mu_k,\Sigma)$. More specifically, what you propose is this

\begin{equation} p(x_i,y_i) = \{ \begin{array}{lc} \pi N(x_i|\mu_1,\Sigma)N(x_i|\mu_0,\Sigma) & \text{when } y_i=1 \\ (1-\pi) N(x_i|\mu_1,\Sigma)N(x_i|\mu_0,\Sigma) & \text{when } y_i=0 \end{array} \end{equation}

In other words, you have to turn on\off $N(x_i|\mu_k,\Sigma)$ at the same time as $p(y_i)$.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.