37
votes
Accepted
Different covariance types for Gaussian Mixture Models
A Gaussian distribution is completely determined by its covariance matrix and its mean (a location in space). The covariance matrix of a Gaussian distribution determines the directions and lengths of ...
- 291k
22
votes
In cluster analysis, how does Gaussian mixture model differ from K Means when we know the clusters are spherical?
Ok, we need to start off by talking about models and estimators and algorithms.
A model is a set of probability distributions, usually chosen because you think the data came from a distribution like ...
- 24.6k
19
votes
Accepted
Finding category with maximum likelihood method
This is a classic unsupervised learning problem that has a simple maximum likelihood solution. The solution is a motivating example for the expectation maximization algorithm. The process is:
...
- 53.3k
17
votes
Accepted
Growing number of Gaussians in a mixture
If your goal is to find the maximum-likelihood mixture of size $n+1$, then you can use the existing solution as an initialization, once you have enlarged it to have one more Gaussian. To enlarge it, ...
- 6,650
17
votes
Accepted
Singularity issues in Gaussian mixture model
If we want to fit a Gaussian to a single data point using maximum likelihood, we will get a very spiky Gaussian that "collapses" to that point. The variance is zero when there's only one point, which ...
- 14.1k
17
votes
Accepted
Why Expectation Maximization is important for mixture models?
In principle, both EM and standard optimization approaches can work for fitting mixture distributions. Like EM, convex optimization solvers will converge to a local optimum. But, a variety of ...
- 29.8k
16
votes
Accepted
Why do we use Gaussian distributions in Variational Autoencoder?
Normal distribution is not the only distribution used for latent variables in VAEs. There are also works using von Mises-Fisher distribution (Hypershperical VAEs [1]), and there are VAEs using ...
- 10.3k
15
votes
Simulate from a truncated mixture normal distribution
Simulation from a truncated normal is easily done if you have access to a proper normal quantile function. For instance, in R, simulating $$
\mathcal{N}_a^b(\mu,\sigma^2)$$where $a$ and $b$ denote the ...
- 92.7k
15
votes
What are Gaussian Scale mixtures? And how to generate samples of Gaussian scale mixture with given scale and location parameter?
The normal (or gaussian) pdf (probability density function) is
$$ \DeclareMathOperator{\E}{E} \DeclareMathOperator{\Var}{Var}
f(x;\mu,\sigma^2) = \frac1{\sqrt{2\pi\sigma^2}}\cdot \exp\left(-\...
- 65.8k
15
votes
Finding category with maximum likelihood method
What you are describing is a mixture of two Gaussians.
$$
f(x) = \pi \, \mathcal{N}(\mu_1, \sigma_1^2) + (1 - \pi) \, \mathcal{N}(\mu_2, \sigma_2^2)
$$
where $\pi \in (0, 1)$ is a mixing proportion. ...
Tim♦
- 115k
14
votes
Why is optimizing a mixture of Gaussian directly computationally hard?
First, GMM is a particular algorithm for clustering, where you try to find the optimal labelling of your $n$ observations. Having $k$ possible classes, it means that there are $k^n$ possible ...
- 13.1k
14
votes
In cluster analysis, how does Gaussian mixture model differ from K Means when we know the clusters are spherical?
In short, $k$-means can be viewed as the limiting case of Expectation-Maximization for spherical Gaussian Mixture Models as the trace of the covariance matrices goes to zero. What follows is a ...
- 1,214
13
votes
Accepted
How to fit mixture model for clustering
Here is script for using mixture model using mcluster.
...
- 2,098
12
votes
What is "mixture" in a gaussian mixture model
A mixture distribution combines different component distributions with weights that typically sum to one (or can be renormalized). A gaussian-mixture is the special case where the components are ...
- 97.4k
12
votes
Gaussian Mixture Models: Maximum Likelihood Estimation or Expectation Maximization?
Mehrin, you are in danger of creating a false dichotomy. EM is an optimisation technique that can be used to find maximum likelihood estimates and so the choice is not "one or the other".
In mixture ...
- 639
12
votes
Accepted
Is it important to make a feature scaling before using Gaussian Mixture Model?
I'm going to assume that you mean , when you say "using a Gaussian Mixture Model", you mean fitting a mixture of (possibly multivariate) Gaussians to some data, for the purposes of ...
- 1,834
12
votes
In cluster analysis, how does Gaussian mixture model differ from K Means when we know the clusters are spherical?
@ThomasLumley's answer is excellent.
For a concrete difference, consider that the only thing you get from $k$-means is a partition. The output from fitting a GMM can include much more than that. For ...
11
votes
Accepted
Mclust model selection
Solution found:
So, to restate the question, why does the Mclust function default to the model with the highest BIC value as the "best" model?
Great question! ...
- 2,220
11
votes
Accepted
Why use a Gaussian mixture model?
I'll borrow the notation from (1), which describes GMMs quite nicely in my opinon. Suppose we have a feature $X \in \mathbb{R}^d$. To model the distribution of $X$ we can fit a GMM of the form
$$f(...
- 2,464
10
votes
Accepted
Generate sample data from Gaussian mixture model
Sampling from mixture distribution is super simple, the algorithm is as follows:
Sample $I$ from categorical distribution parametrized by vector $\boldsymbol{w} = (w_1,\dots,w_d)$, such that $...
Tim♦
- 115k
9
votes
Accepted
Gaussian mixture vs. Gaussian process
To answer your last question, Gaussian process is a discriminative model as opposed to generative. Therefore, you will not be able to model $p(x, y)$ using Gaussian process. Gaussian process models $...
- 925
9
votes
Accepted
Finding the point of maximum probability in a mixture of gaussians
If you have a mixture of $n$ one-dimensional Gaussians, you can have anything between one and $n$ local maxima:
...
- 97.4k
9
votes
Accepted
Fit mixture of distributions to your time-series data in R
There is a misunderstanding in your question that needs a correction. Time-series model is not univariate since you have two variables: actual values and time. To provide an example let's take a time-...
Tim♦
- 115k
9
votes
Accepted
AICc is picking overly complex models - something stricter?
You could try cross-validation, fitting all five models for each fold and picking the model that has the lowest MSE (or whatever other error measure you are interested in) on the holdout folds. That ...
- 97.4k
9
votes
Accepted
A Gaussian Mixture Model Is a Universal Approximator of Densities
The idea is that an arbitrary density on $\mathbb{R}$, $f(\cdot)$, can be approximated by a Gaussian mixture model
$$g_k(\cdot;\boldsymbol{\omega,\mu,\sigma})=\sum_{i=1}^k \omega_i \varphi(\cdot;\...
- 92.7k
9
votes
Why do we use Gaussian distributions in Variational Autoencoder?
We use normal distribution because it is easily reparameterized. Also a sufficiently powerful decoder can map the normal distribution to any other distribution, so from a theoretical viewpoint, the ...
- 23.2k
8
votes
Why is optimizing a mixture of Gaussian directly computationally hard?
In addition to juampa's points, let me signal those difficulties:
The function $l(\theta|S_n)$ is unbounded, so the true maximum is $+\infty$ and corresponds to $\hat\mu^{(i)}=x_1$ (for instance) and ...
- 92.7k
8
votes
If k-means clustering is a form of Gaussian mixture modeling, can it be used when the data are not normal?
GMM uses overlapping hills that stretch to infinity (but practically only count for 3 sigma). Each point gets all the hills' probability scores. Also, the hills are "egg-shaped" [okay, they're ...
- 231
8
votes
How to tell if a mixture of Gaussians will be multimodal
Miguel Carrera-Perpinan has a webpage on this topic with associated software. This does not directly solve your question, but indicates that
unidimensional Gaussian mixtures with $k$ components have ...
- 92.7k
8
votes
Number of Gaussian mixture components needed to approximate any distribution
I am afraid this is an absurd question: there is no magical number and no upper bound on the number of components in a Gaussian mixture for approximating (in which sense?) any distribution. Just think ...
- 92.7k
Only top scored, non community-wiki answers of a minimum length are eligible