Difference between multinomial distribution and a mixture distribution

Question

Forgive me if these two things are completely unrelated but I often find myself confusing one for the other. From my understanding, for example in a Gaussian mixture distribution you mix $n$ Gaussian distributions and each Gaussian is multiplied by a scalar that determines the "amount" that the Gaussian makes up the mixture.

For a multinomial distribution, if I'm rolling a $k$ sided die $n$ times then I get a distribution over the probability of each of the $k$ sides (maybe I have this wrong?).

This at least to me seems like both a mixture and a multinomial consist of multiple distributions. Is the key difference then the scalar factor in the mixture?

Answer 1

If ${\bf X}=(X_1,\cdots,X_k)$ is a Multinomial with $k$ cells, $n$ trials, and probability vector ${\bf p}=(p_1,\cdots,p_k)$, written ${\bf X}\sim\textsf{Mult}_k(n,{\bf p})$, then ${\bf X}$ provides a distribution over the number of trials that fall into each cell. Marginally, each $X_i$ is a (binomial) distribution over the number of trials that fall into cell $i$.

There are two key components to a mixture, a kernel with parameters $\theta$, represented by a pdf $p(x|\theta)$, and a mixing distribution $p(\theta)$. The mixture of the this kernel and mixing distribution is given by $p(x)=\int p(x|\theta)p(\theta)d\theta$. In the case of a $k$ component Gaussian mixture model, the kernel $p(x|\theta)$ is a Gaussian pdf (where $\theta$ can be either the mean, the variance, or both the mean and the wariance if you want to infer them both), and the mixing distribution is the weighted sum of point masses at $k$ values of $\theta$, i.e. $g(\theta)=\sum_{i=1}^{k}\pi_i\delta_{\theta_i}(\theta)$, where $\delta_{\theta_i}(\cdot)$ is a point mass at $\theta_i$, the parameter of the $ith$ component of the mixture, and $\pi_i$ is the mixing proportion of the $ith$ component (where the $\pi_i$ add up to 1 so that $g$ is a valid pdf). You can verify through properties of integrating over point masses/dirac delta functions that this leads to the standard representation of a $k$ component Gaussian mixture: $p(x)=\sum_{i=1}^{k}\pi_ip(x|{\theta_i})$.

If you'd like, you can think of a Multinomial with $n=1$ as a mixture of point masses at each side of the die, where each point mass has a weight equal to the probability of rolling that side. That is, if you have a fair die, so $p_i=1/6$ for each side $i\in\{1,\cdots,6\}$, we can represent the random variable $X$, the side the Multinomial comes up as, as either $$X\sim\textsf{Mult}_6(1,p_1,\cdots,p_6)$$ (this is non-standard, it's usually written as a random vector as above), or as $$X\sim\sum_{i=1}^{6}p_i\delta_{i}(\cdot).$$ I'm not exactly sure what the kernel would be in this mixture though.

Answer 2

In essence, the difference is that multinomial is sum of (catagorical) independent random variable, while mixture model is not (well, that's why it has a term).

An interesting property of finite mixture models is that the density function is a weighted sum of the densities of the underline distributions. This often creates a multimodal distribution. (For non-finite mixture models, see @marmle answer: https://stats.stackexchange.com/a/319950/86267)

Sum of independent random variables can create a distribution that is completely different. Consider the following two cases:

• Sum of Bernoulli random variable creates a binomial random variable.
• Sum of Gaussian random variable creates a Gaussian random variable.

Calculating the densities in this cases might be challenging. Two common techniques are convolution and multiplying moment generating functions.

Answer 3

Let $p_1, \ldots, p_K$ be numbers between $0$ and $1$ which add up to $1$. Now, let $X$ be a random variable with the following property: $X=j$ with probability $p_j$ for each $j=1,\ldots,K$. Then, $X$ has a categorical distribution. Suppose $X_1,\ldots,X_n$ are sampled independently $n$ times and $\textbf{V}$ is the vector which counts the number of times each category occurs. That is, $\textbf{V}=[V_1,\ldots,V_K]^\top$ where $V_1=\sum_{i=1}^K I(X_i=1), \ldots, V_K=\sum_{i=1}^K I(X_i=K)$. Then, $\textbf{V}$ has a multinomial distribution.

Now, suppose $Z_1,\ldots,Z_K$ are $K$ different random variables with (possibly distinct) Gaussian distributions. Let $Y$ be yet another random variable with the following property: $Y=Z_j$ with probability $p_j$ for each $j=1,\ldots,K$. Then, $Y$ has a Gaussian mixture distribution.