GMM(Gaussian Mixture Model) itself is a mixture of Gaussian with each having the proportion of $\pi_k$, $$\sum_{k=1}^{K}\pi_k=1$$this is easy to understand. But when introducing the latent, I don't understand it, because there can only be one $z_k=1$, then how this is related to the soft assignment in GMM model?
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
2
It may help if you can define the $z$ variable a bit more thoroughly in your post, since people can use this variable for different things.
However I am going to assume that you are referring to $z$ as a latent variable that randomly selects from a one-hot encoded vector which Gaussian to choose, for example when randomly sampling from the GMM.
For example, when randomly sampling a GMM you need to pick "one out of $K$" Gaussians to pull the sample from. How do you automate that process? You define an extra latent variable, $z$ to act as the middle man in a sense which communicate which Gaussian to pick from a one hot encoded vector of choices.
So for some data point, $X_i$, we can ask the question what is the probability distribution of $X_i$ assuming it belongs to class "2", so it allows to formalize questions such as: $P(X_i | Z = 2)?$, to which we can allocate its Gaussian $P(X_i | Z = 2) = \mathcal{N}(\mu_2,\sigma^2_2)$, and then we can assign multinomial priors to each cluster and even as $P(Z=2)$, what is the prior that we belong to class 2. And so on.
So it's a latent middle man that helps with the random choice of which Gaussian to pick in the mixture when sampling, or calculting the EM algorithm etc.
See also this link: https://stats.stackexchange.com/a/380776/117574
-
$\begingroup$ Thank you for your answer, it is very helpful. I have another question, in M-step, the GMM is maximizing the log likelihood of complete data $argmax_{\theta}\int p(z|x,\theta^{old})lnp(x,z|\theta)dz$, is is universal for all EM problems even if it is not gaussian mixture models? $\endgroup$ May 26 at 7:27
-
$\begingroup$ Correct. Read here and notice that the problem is set up in a very general way stats.stackexchange.com/a/371087/117574 and it is only until the author says "For the normal mixture" or something like that, is when he can only start to specify how to expand these expressions for GMMs in particular. Otherwise EM is a generic algorithm. Imagine if in a vacuum you gave me your integral expression --- I would have no clue it is related to Gaussians in any way. It is simply some generic probabilistic expression. $\endgroup$ May 27 at 5:45
$\begingroup$
$\endgroup$
1
With GMM do you mean "growth mixture model?" If so, then it looks like the proportion π_k might indicate the size of a particular latent class (class size parameter). Since the K latent classes in GMM are exhaustive and mutually exclusive, the π_k parameters for all classes add up to 1.0. This is what the equation states. (For each specific latent class, you should get a specific π_k parameter estimate that is < 1.)
-
$\begingroup$ Sorry about the confusion, GMM means Gaussian Mixture Model here. I added it to the question description. $\endgroup$ May 26 at 10:59