2
$\begingroup$

I've recently learned about Gaussian mixture models (GMM) in school. The formula for a GMM goes something like this:

$$p(x) = \sum_{k=1}^{K} \pi_i\mathcal{N}(x|\mu_i,\,\sigma^{2}_i)$$

Now, I know from probability theory that given a certain sample $x$, the probability for it should sum to 1.

For example, if I roll a six-sided die, the probability that I either get or don't get the number $5$ is equal to 1.

So what exactly does $p(x)$ here represent? The probability that a certain $x$ falls in the model? I feel like I'm missing something fundamental here.

$\endgroup$

2 Answers 2

2
$\begingroup$

The first this is that the $p(x)$ represents a density and not a probability in the classical way that an event $x$ is likely to happen with probability $p(x)$.

Then the correct way to think of $p(x)$ is as the weighted sum of densities on a given point $x$. With that said think about the following example:

You have a Normal distribution centred around $-1$ and a Normal distribution centred around $1$. Now someone tells you that a point $x$ is likely to belong to the first normal distribution with probability $\pi =0.2$ then the probability for the point to belong to the second is $1-\pi =0.8$. With that in mind then someone ask you what is the density on the point $x=0$, this naturally will be the sum of both normal distributions on the point $x=0$. But is it only that? You have the information that the second is 4 times more likely than the first one. So, you would expect the density of the second normal distribution to contribute 4 times more than the first one. Hence, you will have that the density on a particular point $x=0$ will be $p(0) = 0.2 * Norm(0;-1,\sigma) + 0.8*Norm(0;1,\sigma)$

And again is not a probability is a density, in discrete case this changes. Hope that helps

$\endgroup$
3
$\begingroup$

The formula says that the random variable $X$ in question has probability density function $p(x)$ given by the weighted sum of $K$ different Gaussian densities, with weights summing to one.

The rationale behind a GMM is that, somehow, we believe your observations come from $K$ different groups, with each group having its own distribution. Then you leave the liberty to each observation to belong to any of each group. Thus you can think of data generated by a mixture model in two steps:

(1) draw group $i$ with probability $\pi_i$,

(2) draw an observation from the distribution of the group in (1).

Novices typically confuse the distribution of the mixture model with that of the sum of random variables, say $X_1+X_2+\cdots+X_K$. But, for instance, a sum of $K$ Gaussian random variables, $X_1+\cdots+X_K$, is still a Gaussian, whereas a random variable whose density is the sum of Gaussian densities (as the case in question) is not Gaussian.

$\endgroup$
6
  • $\begingroup$ I get that. But I still don't understand how p(x) represents a probability. What does p(x) even mean? Also what you're getting to is p(z|x) I think right? The probability that an x belongs to a certain cluster. $\endgroup$
    – Dude156
    Commented Oct 16, 2022 at 20:55
  • 1
    $\begingroup$ (1) $p(x)$ is just the density at $x$, as for any continuous random variable. The density, as you may recall, is not a probability statement when $X$ is continuous. (2), yes, it's a conditional probability statement. In other terms, $p(x)$ gives you a stochastic representation for your samples when they are thought to belong to (possibly) different clusters; each sample may belong to the $i$th cluster with probability and if it belongs to it, it has distribution $N(\mu_i,\sigma_i^2).$ $\endgroup$
    – utobi
    Commented Oct 16, 2022 at 21:14
  • $\begingroup$ Ah, and 1 - p(x) then represents the probability that the samples belong to no cluster correct? $\endgroup$
    – Dude156
    Commented Oct 16, 2022 at 21:18
  • $\begingroup$ No, p(x) IS NOT a probability. Each observation will necessarily belong to one of the clusters as $\sum_i \pi_i=1$. $\endgroup$
    – utobi
    Commented Oct 16, 2022 at 21:22
  • $\begingroup$ But then why doesn't p(x) = 1 summed over the marginals? Is it because it's a density and the density at a specific point is always 0? But what about values near x? Shouldn't that result in some non zero number? I'm very confused here. Thanks for the help btw :) Also I get why p(z|x) = 1. But that's because we normalize it as such. $\endgroup$
    – Dude156
    Commented Oct 16, 2022 at 21:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.