p(x) in Gaussian mixture model I've recently learned about Gaussian mixture models (GMM) in school. The formula for a GMM goes something like this:
$$p(x) = \sum_{k=1}^{K} \pi_i\mathcal{N}(x|\mu_i,\,\sigma^{2}_i)$$
Now, I know from probability theory that given a certain sample $x$, the probability for it should sum to 1.
For example, if I roll a six-sided die, the probability that I either get or don't get the number $5$ is equal to 1.
So what exactly does $p(x)$ here represent? The probability that a certain $x$ falls in the model? I feel like I'm missing something fundamental here.
 A: The formula says that the random variable $X$ in question has probability density function $p(x)$ given by the weighted sum of $K$ different Gaussian densities, with weights summing to one.
The rationale behind a GMM is that, somehow, we believe your observations come from $K$ different groups, with each group having its own distribution. Then you leave the liberty to each observation to belong to any of each group. Thus you can think of data generated by a mixture model in two steps:
(1) draw group $i$ with probability $\pi_i$,
(2) draw an observation from the distribution of the group in (1).
Novices typically confuse the distribution of the mixture model with that of the sum of random variables, say $X_1+X_2+\cdots+X_K$. But, for instance, a sum of $K$ Gaussian random variables, $X_1+\cdots+X_K$, is still a Gaussian, whereas a random variable whose density is the sum of Gaussian densities (as the case in question) is not Gaussian.
A: The first this is that the $p(x)$ represents a density and not a probability in the classical way that an event $x$ is likely to happen with probability $p(x)$.
Then the correct way to think of $p(x)$ is as the weighted sum of densities on a given point $x$. With that said think about the following example:
You have a Normal distribution centred around $-1$ and a Normal distribution centred around $1$. Now someone tells you that a point $x$ is likely to belong to the first normal distribution with probability $\pi =0.2$ then the probability for the point to belong to the second is $1-\pi =0.8$. With that in mind then someone ask you what is the density on the point $x=0$, this naturally will be the sum of both normal distributions on the point $x=0$. But is it only that? You have the information that the second is 4 times more likely than the first one. So, you would expect the density of the second normal distribution to contribute 4 times more than the first one. Hence, you will have that the density on a particular point $x=0$ will be $p(0) = 0.2 * Norm(0;-1,\sigma) + 0.8*Norm(0;1,\sigma)$
And again is not a probability is a density, in discrete case this changes. Hope that helps
