4
$\begingroup$

I am trying to understand a little better mixtures of Gaussians and their generative approach in general. For a mixture of Gaussians we start with this formula: $$p(x)=\sum_{k=1}^{K}\pi_{k}\cdot N(x|\mu_{k},\Sigma_{k})\,,\quad \sum_{i=1}^K\pi_k=1\,,$$ where $\mu_{k}$ and $\Sigma_{k}$ are the means and covariances for each Gaussian.

Then we try to do maximum likelihood on this function and run the EM algorithm. But I don't understand the meaning of the parameter $(\pi_{k})_{1\le k\le K}$. Are they just the probability with which we choose one of the Gaussians in the above formula or far from it?

Also I would like to ask about the generative approach in clustering. By generative we mean that we have a model that can generate data. In Bishop's book he says that "by sampling from them (meaning generative models) we can generate synthetic data points in the input space".

$\endgroup$
4
$\begingroup$

To answer your first question, you can think of the proportion $ \pi_k $ as the probability that an observation comes from population k (Gaussian with mean $ \mu_k $ and covariance $ \Sigma_k^2$). For sampling, you first sample from a uniform to decide from which Gaussian you will sample each observation. For estimation, you decide what the proportions are (E step) before maximizing likelihood (M step), and then iterate.

$\endgroup$
  • 1
    $\begingroup$ EM is only one approach to one estimation method, among others (moments, Bayesian, non-parametric...) $\endgroup$ – Xi'an Dec 13 '14 at 18:59
  • $\begingroup$ @Xi'an In some fields it's also common to simply employ brute force numerical maximization (usually when the separate distributions are more complicated than Gaussians). $\endgroup$ – jwimberley Jan 11 '17 at 3:05
6
$\begingroup$

Is it just the probability with which we choose one of the Gaussians in the above formula or far from it?

Yes, pretty much.

By generative we mean that we have a model that can generate data?

Yes. In this case use the $\{\pi_j\}$ to randomly choose a $k$ then draw $x \sim \mathcal N(\mu_k,\Sigma_k)$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.