# Mixture of Multinomials

I have implemented a Naive Bayes Model for a mixture of independent Bernoulli. Where the conditional probability can be written as:

$$\mathbb{P}(Y=j | X) \propto \omega_{j} \prod_{i=1}^{d} \mu_{i, j}^{x_{i}}\left(1-\mu_{i, j}\right)^{1-x_{i}}$$

Where $$\mu_{i, j}=\mathbb{P}\left(x_{i}=1 | Y=j\right)$$ and $$\omega_{h} :=\mathbb{P}(Y=h)$$ where $$Y$$ is the hidden variable and $$x_{i}$$ are the observables. The parameters of the model are learned via Expectation-Maximization.

I wanted to generalized this model to non-binary observables by considering a mixture of multinomials. How should I define the conditional probability?

Since the observables in the original problem are binary you just need one parameter, $$\mu_{i,j}$$, for each observable conditioning on a given y and the other one can be got by $$1-\mu_{i,j}$$.
When it comes to multinomial observables that would be K-1 where K is the number of classes of the observation variable. Thus, you can change $$\mathbb{P}(Y=j | X) \propto \omega_{j} \prod_{i=1}^{d} \mu_{i, j}^{x_{i}}\left(1-\mu_{i, j}\right)^{1-x_{i}}$$ to $$\mathbb{P}(Y=j | X) \propto \omega_{j} \prod_{i=1}^{d} \prod_{k=1}^K\mu_{i, j,k}1\{x_i=k\}$$.
And $$\mu_{i, j,k}=\mathbb{P}\left(x_{i}=k | Y=j\right)$$ where $$x_i$$ can take k values and $$k>2$$, and $$1\{x_i=k\}$$ equals 1 if $$x_i$$ is the kth class and 0 otherwise.