3
$\begingroup$

When applying the M-step for a mixture of Bernoulli distributions, one of the parameters in our maximization is the Bernoulli parameter $\mu_{k}$, where $k$ is the index of the "mixture component", and $$ p(x|\mu_k) = \prod_{i=1}^D \mu_{ki}^{x_{i}}(1-\mu_{ki})^{(1-x_{i})}. $$ In our maximization with respect to this parameter, we get the following expression $$ \begin{align} \frac{\partial}{\partial \mu_{ki}}\mathbb{E}_{Z}[\ln p(X, Z | \mu, \pi) &= \sum_{n=1}^N \langle z_{nk} \rangle \left( \frac{x_{ni}}{\mu_{ki}} - \frac{1 - x_{ni}}{1 - \mu_{ki}} \right) \\ &= \frac{\sum_n \langle z_{nk} \rangle x_{ni} - \sum_n \langle z_{nk} \rangle \mu_{ki} }{\mu_{ki}(1-\mu_{ki})} \end{align} $$ where $$ \langle z_{nk} \rangle = p(z_{nk} | x_n, \mu_k, \pi_k) $$ Obviously, setting this to zero and solving for $\mu_{ki}$, we get the standard solution $$ \mu_{ki} = \frac{\sum_n \langle z_{nk} \rangle x_{ni}}{\sum_n \langle z_{nk} \rangle} $$

With that in mind, my question is as follows. Isn't there a constraint on $\mu$ such that $\sum_i \mu_{ki} = 1$? If so, then why is this not included in the maximization; i.e. why don't we formulate the Lagrangian which includes the term $ \lambda \left(\sum_i \mu_{ki} - 1\right) $?

$\endgroup$
  • $\begingroup$ If I'm understanding this right, you are looking at mixtures of distributions on $\{0, 1\}^D$ (which are restricted to be independent within cluster); I wouldn't call this a "mixture of Bernoulli distributions" since a mixture of Bernoulli distributions is trivially just another Bernoulli distribution. "Mixture of multivariate Bernoulli distributions" is a more appropriate name. $\endgroup$ – guy Aug 10 '14 at 18:29
3
$\begingroup$

You would need that constraint if you were working with a multinomial distribution. Multinomial distribution are used when you have K possible outcomes. They are coded in a 1-of-K fashion, that is, a vector such that only one of its component is non-zero (usually 1). In that case, $$ p(x|\mu) = \prod_{k=1}^{K}\mu_{k}^{x_{k}} $$ which implies that, $$ \sum_{x}p(x|\mu) = \sum_{k}\mu_{k} = 1 $$

Now, you have a vector ,$x$, such that each component follows a Bernoulli distribution, that is, they are independent from each other. There you code whether a given feature is present or not.

$\endgroup$
  • $\begingroup$ Thanks for the answer, but I'm still confused. Are you saying that the Bernoulli does not have the constraint I mentioned in my post? $\endgroup$ – Just Aug 12 '14 at 14:52
  • $\begingroup$ Sorry for the late answer. The first expression you write embodies the assumption that the components of the $x$ vector are independent from each other, so the constraint does not apply. $\endgroup$ – jpmuc Aug 16 '14 at 12:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.