Looking at this link on Gaussian Mixtures and EM: http://www.ics.uci.edu/~smyth/courses/cs274/notes/EMnotes.pdf

from the link:

Given a data set D = {$x_1, x_N$} where $x_i$ is a d-dimesional vector. Assume that the points are generated from a density p(x). Further assume p(x) is a finite mixture model with K components:

p(x|$\theta$) = $\sum_1^K \alpha_kp_k(x|z_k,\theta_k)$

where $\alpha_k$ are the mixture weights. $\sum_{k=1}^K\alpha_k=1$

the author then says on page 2 under Gaussian Mixture Models" We can define a gaussian mixture model by making each of the k components a gaussian density with parameters $\mu_k$ and $\sum_k$. Each component is a multivariate gaussian density.

My question is: Is the last statement correct? Is each component multivariate gaussian?

Here is an example:

I have a 1-dimensional vector, X, with 2 components:


I was thinking for gaussian mixtures each observation in the X vector is either from component 1 or 2. So the first entry .67 would be either from

set up #1

k=1 ~ N($\mu_1,\sigma_1$) or

k=2 ~ N($\mu_2,\sigma_2$)

However, the author is saying .67 is either from 2 multivariate distributions:

set up #1

k=1 ~ N($\mu_1^1, \mu_2^1; \sum_1$) or

k=2 ~ N($\mu_1^2, \mu_2^2; \sum_2$)

So given the example above what is the correct setup #1 or #2?

Just thinking about this in the context of the EM algorithm to estimate the parameters of the model I think it is #1 because at each iteration of the EM algorithm you are estimating the parameters of a single multivariate distribution.


You seem to confuse sample and observations: if one takes $n$ observations from a $p$-dimensional mixture distribution, each observation is a $p$-dimensional vector. It is associated with a latent variable that takes as value 1,2,... depending on which component the vector is generated from.

To make the distinction even clearer, here is a sample of 250 points generated from a univariate mixture$$0.3\mathcal{N}(0,1)+0.7\mathcal{N}(2,1)$$

enter image description here

generated as


and here s a sample of 250 points generated from a bivariate mixture$$0.3\mathcal{N}_2(0_2,I_2)+0.7\mathcal{N}_2((6,-3),i)$$

enter image description here

generated as


In the first case, the sample starts with

> data[1:10]
 [1]  1.1093865  3.9148698  0.7089910  0.8179517  2.1479357  1.1273827
 [7]  1.9081127  1.5963952  3.0439887 -0.7480628

and in the second case it begins as

> dota[1:10,]
           [,1]       [,2]
 [1,]  4.082637 -2.3638552
 [2,]  6.417835 -1.9432360
 [3,]  5.619073 -0.8438118
 [4,]  6.704312 -3.1426273
 [5,]  6.000480 -2.8611720
 [6,]  5.370948 -2.0932625
 [7,] -1.202043  0.7533113
 [8,] -0.384621 -0.4188970
 [9,]  6.910961 -1.3184499
[10,]  4.453260 -2.9979343
  • $\begingroup$ Yes the .67 will have a latent variable,z, that is 1 or 2 indicating if .67 came from distribution 1 or 2. My question is are the distributions,1 &2, multivariate or univarite....setup #1 or #2 $\endgroup$ – user3022875 Nov 7 '16 at 15:50
  • $\begingroup$ If 0.67 is one observation this is a one-dimension mixture (setup #1). If 0.67 is a component of an observation vector, for instance, (0.67,2.2), this is a two-dimensional mixture (setup #2). $\endgroup$ – Xi'an Nov 7 '16 at 16:11
  • $\begingroup$ what if the observation vector has 100 term (.67, 2.2,1.5.......) then you would say there are a 100 dimensional mixture with k=1 ~ N(mu1,mu2, ...mu100, covariace matrix 100x100)? In my example I state this is 1 dimensional vector, X, with 2 components. Components are distributions. $\endgroup$ – user3022875 Nov 7 '16 at 16:18
  • $\begingroup$ It doesn't matter how many observations X has, 1 or 2 or 100, each observation will only come from 1 of 2 distributions, k=1 or k=2. the distributions do not change from being univariate to multivariate based on the length of X. my question is given 1 vector, i.e. 1 D, and 2 components are the components univariate or multivariate. The author says multivariate and with k=2, that would mean in the EM algorithm you'd have to estimate 4 means and 2 covariance matricies each time through the loop. I believe you only estimate 2 means and 1 covariance matrix in the loop. $\endgroup$ – user3022875 Nov 7 '16 at 16:21
  • $\begingroup$ The meaning of X itself is unclear. Let me repeat my answer: the dimension of the mixture is the dimension of each observation in the sample. $\endgroup$ – Xi'an Nov 7 '16 at 16:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.