What's the difference between Multivariate Gaussian and Mixture of Gaussians?

What's the difference between Multivariate Gaussian and Mixture of Gaussians?

If I have a Multivariate Gaussian and making all the data into ONE vector, is that a Mixture of Gaussians in 1 dimension?

On the other hand, if I have a mixture of Gaussian (say two models and the mixing ratio is 0.3,0.7), Can I make it to 2D Gaussian?

• no, not at all. – Xi'an Dec 21 '17 at 20:19
• if you take a random sample from the vector, it follows a gaussian mixture distribution? – Lii Dec 21 '17 at 20:44
• Multi-modality. In the case of the Gaussian mixture, the camel can have two (or n) humps. Stacking the data does not reduce to a single dimension -- a topologist would have a field day with an assertion like that, and that should intimidate you because those guys start out with "objects" made of infinitely stretchable elastic. – EngrStudent Dec 21 '17 at 21:06

We can write out a $k$ dimensional Multivariate Gaussian as $${\bf X}=(X_1,\cdots,X_k)\sim\textsf{MVN}(\mu,\Sigma),$$ where $\mu=(\mu_1,\cdots,\mu_k)$ is the mean vector, and $\Sigma$ is the positive definite $k\times k$ covariance matrix.

We can write out a $k$ component mixture of (1 dimensional) Gaussians as $$X\sim \sum_{i=1}^k\pi_i\textsf{Norm}(\mu_i,\sigma_i^2),$$ where $\pi_i$ is the mixing proportion of the $i$th component and $(\mu_i,\sigma_i^2$) are the parameters of the $i$th component. Alternatively, using a latent variable $z$, we can write the mixture as $$X\sim \textsf{Norm}(\mu_z,\sigma_z^2),\ z\sim\textsf{Categorical}_k(\pi_1\cdots,\pi_k).$$

As you can see, the Multivariate Gaussian is defined as a $k$ dimensional random vector, and the mixture of Gaussians is defined as a random variable (which you can call a 1 dimensional random vector). The dimension of your data in a mixture of Gaussians is determined by the dimensions of the Gaussians you're mixing. There's no general connection between the two, as you can have, for example, multimodal mixtures, whereas Gaussians can only be unimodal.

I do not intend to be rigorous here I just want to give you the intuition. You can go deeper on this subject both reading textbooks or googling it.

If you have one random variable $X$ you say that it follows a normal distribution if ...

the density $f_X(x) = \frac{1}{\sqrt{2\pi\sigma^2}}e^{(...)}$

Suppose now you have several random variables $X_1, X_2, ..., X_n$

They form a vector $\textbf{X}$ and you say that $\textbf{X}$ is Multivariate Gaussain if the joint density $f_{\textbf{X}}(\textbf{x})$ has a specific form. This is just an extension of the definition we just saw in the univariate case.

We are referring to the joint probability here $P(X_1 \le x_1, X_2 \le x_2,...,X_n \le x_n)$ that is a multidimensional integral of the joint density $f_{\textbf{X}}(\textbf{x})$

The Mixture of gaussian is a different concept and we are talking about hierarchical models here

Suppose that the random variable $X$ has two parameters $\mu$ and $\sigma$, but both of them are also random with $\mu \sim N (m_1, s_1)$ and $\sigma \sim N (m_2, s_2)$

it could be that the conditional distribution of $X$ given $\mu$ and $\sigma$ is normal. In tis case you write $X \mid \mu, \sigma \sim N(\mu, \sigma)$ and that is a mixture of gaussian

• That is not what is generally meant by a mixture model. – Dilip Sarwate Dec 22 '17 at 3:44
• (-1) Hierarchical models and Mixture models are not the same things. – TenaliRaman May 30 '18 at 10:51
• @TenaliRaman That is not true. I studied on Casella Berger and on chapter 4, par 4.4, page 162-165 they explicitly talks about Hierarchical Model and Mixture distribution. " .. $X$ is said to have a mixture distribution if the distribution of $X$ depends on a quantity that also has a distribution." Following this argument I correclty described a mixture of gaussians, unless you want to contradict one of the most used statistics textbooks in the entire world. Please remove the downvote. – Hard Core May 30 '18 at 14:15

If I have a Multivariate Gaussian and making all the data into ONE vector, is that a Mixture of Gaussians in 1 dimension?

If the components of the multivariate Gaussian are uncorrelated then it will be a mixture of Gaussians, but this is not a good way to think about mixture Gaussian models.

On the other hand, if I have a mixture of Gaussian (say two models and the mixing ratio is 0.3,0.7), Can I make it to 2D Gaussian?

I'll answer this with an example which I find illuminating. Suppose that you measure the heights of people from nation A and nation B, but you don't know which nation a given person belongs, so you only know the heights. If the heights are coming from distributions $X_A \sim \mathcal{N}(\mu_A, \sigma_A)$ and $X_B \sim \mathcal{N}(\mu_B, \sigma_B)$ in proportions $\pi_A$ and $\pi_B$, than the density you will see is $$f(x) = \pi_A \mathcal{N}(x | \mu_A, \sigma_A) + \pi_B \mathcal{N}(x | \mu_B, \sigma_B),$$ which is a bimodal distribution.

So, if you don't know the corresponding nationality for a given height data, you have no way of modeling this as a multivariate Gaussian $(X_A, X_B)$. Thus, the answer to your second question is no, in general. Of course, if someone hands you all necessary data, you can do it, but in this case it makes no practical sense to use mixture distributions to begin with.