37 votes
Accepted

Different covariance types for Gaussian Mixture Models

A Gaussian distribution is completely determined by its covariance matrix and its mean (a location in space). The covariance matrix of a Gaussian distribution determines the directions and lengths of ...
user avatar
  • 291k
22 votes

In cluster analysis, how does Gaussian mixture model differ from K Means when we know the clusters are spherical?

Ok, we need to start off by talking about models and estimators and algorithms. A model is a set of probability distributions, usually chosen because you think the data came from a distribution like ...
user avatar
19 votes
Accepted

Finding category with maximum likelihood method

This is a classic unsupervised learning problem that has a simple maximum likelihood solution. The solution is a motivating example for the expectation maximization algorithm. The process is: ...
user avatar
  • 53.3k
17 votes
Accepted

Growing number of Gaussians in a mixture

If your goal is to find the maximum-likelihood mixture of size $n+1$, then you can use the existing solution as an initialization, once you have enlarged it to have one more Gaussian. To enlarge it, ...
user avatar
  • 6,650
17 votes
Accepted

Singularity issues in Gaussian mixture model

If we want to fit a Gaussian to a single data point using maximum likelihood, we will get a very spiky Gaussian that "collapses" to that point. The variance is zero when there's only one point, which ...
user avatar
  • 14.1k
17 votes
Accepted

Why Expectation Maximization is important for mixture models?

In principle, both EM and standard optimization approaches can work for fitting mixture distributions. Like EM, convex optimization solvers will converge to a local optimum. But, a variety of ...
user avatar
  • 29.8k
16 votes
Accepted

Why do we use Gaussian distributions in Variational Autoencoder?

Normal distribution is not the only distribution used for latent variables in VAEs. There are also works using von Mises-Fisher distribution (Hypershperical VAEs [1]), and there are VAEs using ...
user avatar
  • 10.3k
15 votes

Simulate from a truncated mixture normal distribution

Simulation from a truncated normal is easily done if you have access to a proper normal quantile function. For instance, in R, simulating $$ \mathcal{N}_a^b(\mu,\sigma^2)$$where $a$ and $b$ denote the ...
user avatar
  • 92.7k
15 votes

What are Gaussian Scale mixtures? And how to generate samples of Gaussian scale mixture with given scale and location parameter?

The normal (or gaussian) pdf (probability density function) is $$ \DeclareMathOperator{\E}{E} \DeclareMathOperator{\Var}{Var} f(x;\mu,\sigma^2) = \frac1{\sqrt{2\pi\sigma^2}}\cdot \exp\left(-\...
user avatar
15 votes

Finding category with maximum likelihood method

What you are describing is a mixture of two Gaussians. $$ f(x) = \pi \, \mathcal{N}(\mu_1, \sigma_1^2) + (1 - \pi) \, \mathcal{N}(\mu_2, \sigma_2^2) $$ where $\pi \in (0, 1)$ is a mixing proportion. ...
user avatar
  • 115k
14 votes

Why is optimizing a mixture of Gaussian directly computationally hard?

First, GMM is a particular algorithm for clustering, where you try to find the optimal labelling of your $n$ observations. Having $k$ possible classes, it means that there are $k^n$ possible ...
user avatar
  • 13.1k
14 votes

In cluster analysis, how does Gaussian mixture model differ from K Means when we know the clusters are spherical?

In short, $k$-means can be viewed as the limiting case of Expectation-Maximization for spherical Gaussian Mixture Models as the trace of the covariance matrices goes to zero. What follows is a ...
user avatar
  • 1,214
13 votes
Accepted

How to fit mixture model for clustering

Here is script for using mixture model using mcluster. ...
user avatar
  • 2,098
12 votes

What is "mixture" in a gaussian mixture model

A mixture distribution combines different component distributions with weights that typically sum to one (or can be renormalized). A gaussian-mixture is the special case where the components are ...
user avatar
12 votes

Gaussian Mixture Models: Maximum Likelihood Estimation or Expectation Maximization?

Mehrin, you are in danger of creating a false dichotomy. EM is an optimisation technique that can be used to find maximum likelihood estimates and so the choice is not "one or the other". In mixture ...
user avatar
12 votes
Accepted

Is it important to make a feature scaling before using Gaussian Mixture Model?

I'm going to assume that you mean , when you say "using a Gaussian Mixture Model", you mean fitting a mixture of (possibly multivariate) Gaussians to some data, for the purposes of ...
user avatar
  • 1,834
12 votes

In cluster analysis, how does Gaussian mixture model differ from K Means when we know the clusters are spherical?

@ThomasLumley's answer is excellent. For a concrete difference, consider that the only thing you get from $k$-means is a partition. The output from fitting a GMM can include much more than that. For ...
user avatar
11 votes
Accepted

Mclust model selection

Solution found: So, to restate the question, why does the Mclust function default to the model with the highest BIC value as the "best" model? Great question! ...
user avatar
  • 2,220
11 votes
Accepted

Why use a Gaussian mixture model?

I'll borrow the notation from (1), which describes GMMs quite nicely in my opinon. Suppose we have a feature $X \in \mathbb{R}^d$. To model the distribution of $X$ we can fit a GMM of the form $$f(...
user avatar
  • 2,464
10 votes
Accepted

Generate sample data from Gaussian mixture model

Sampling from mixture distribution is super simple, the algorithm is as follows: Sample $I$ from categorical distribution parametrized by vector $\boldsymbol{w} = (w_1,\dots,w_d)$, such that $...
user avatar
  • 115k
9 votes
Accepted

Gaussian mixture vs. Gaussian process

To answer your last question, Gaussian process is a discriminative model as opposed to generative. Therefore, you will not be able to model $p(x, y)$ using Gaussian process. Gaussian process models $...
user avatar
  • 925
9 votes
Accepted

Finding the point of maximum probability in a mixture of gaussians

If you have a mixture of $n$ one-dimensional Gaussians, you can have anything between one and $n$ local maxima: ...
user avatar
9 votes
Accepted

Fit mixture of distributions to your time-series data in R

There is a misunderstanding in your question that needs a correction. Time-series model is not univariate since you have two variables: actual values and time. To provide an example let's take a time-...
user avatar
  • 115k
9 votes
Accepted

AICc is picking overly complex models - something stricter?

You could try cross-validation, fitting all five models for each fold and picking the model that has the lowest MSE (or whatever other error measure you are interested in) on the holdout folds. That ...
user avatar
9 votes
Accepted

A Gaussian Mixture Model Is a Universal Approximator of Densities

The idea is that an arbitrary density on $\mathbb{R}$, $f(\cdot)$, can be approximated by a Gaussian mixture model $$g_k(\cdot;\boldsymbol{\omega,\mu,\sigma})=\sum_{i=1}^k \omega_i \varphi(\cdot;\...
user avatar
  • 92.7k
9 votes

Why do we use Gaussian distributions in Variational Autoencoder?

We use normal distribution because it is easily reparameterized. Also a sufficiently powerful decoder can map the normal distribution to any other distribution, so from a theoretical viewpoint, the ...
user avatar
  • 23.2k
8 votes

Why is optimizing a mixture of Gaussian directly computationally hard?

In addition to juampa's points, let me signal those difficulties: The function $l(\theta|S_n)$ is unbounded, so the true maximum is $+\infty$ and corresponds to $\hat\mu^{(i)}=x_1$ (for instance) and ...
user avatar
  • 92.7k
8 votes

If k-means clustering is a form of Gaussian mixture modeling, can it be used when the data are not normal?

GMM uses overlapping hills that stretch to infinity (but practically only count for 3 sigma). Each point gets all the hills' probability scores. Also, the hills are "egg-shaped" [okay, they're ...
user avatar
8 votes

How to tell if a mixture of Gaussians will be multimodal

Miguel Carrera-Perpinan has a webpage on this topic with associated software. This does not directly solve your question, but indicates that unidimensional Gaussian mixtures with $k$ components have ...
user avatar
  • 92.7k
8 votes

Number of Gaussian mixture components needed to approximate any distribution

I am afraid this is an absurd question: there is no magical number and no upper bound on the number of components in a Gaussian mixture for approximating (in which sense?) any distribution. Just think ...
user avatar
  • 92.7k

Only top scored, non community-wiki answers of a minimum length are eligible