# Gaussian Mixture description

Looking at this link on Gaussian Mixtures and EM: http://www.ics.uci.edu/~smyth/courses/cs274/notes/EMnotes.pdf

Given a data set D = {$x_1, x_N$} where $x_i$ is a d-dimesional vector. Assume that the points are generated from a density p(x). Further assume p(x) is a finite mixture model with K components:

p(x|$\theta$) = $\sum_1^K \alpha_kp_k(x|z_k,\theta_k)$

where $\alpha_k$ are the mixture weights. $\sum_{k=1}^K\alpha_k=1$

the author then says on page 2 under Gaussian Mixture Models" We can define a gaussian mixture model by making each of the k components a gaussian density with parameters $\mu_k$ and $\sum_k$. Each component is a multivariate gaussian density.

My question is: Is the last statement correct? Is each component multivariate gaussian?

Here is an example:

I have a 1-dimensional vector, X, with 2 components:

X
.67
.9
2.2
5.1
....


I was thinking for gaussian mixtures each observation in the X vector is either from component 1 or 2. So the first entry .67 would be either from

set up #1

k=1 ~ N($\mu_1,\sigma_1$) or

k=2 ~ N($\mu_2,\sigma_2$)

However, the author is saying .67 is either from 2 multivariate distributions:

set up #1

k=1 ~ N($\mu_1^1, \mu_2^1; \sum_1$) or

k=2 ~ N($\mu_1^2, \mu_2^2; \sum_2$)

So given the example above what is the correct setup #1 or #2?

Just thinking about this in the context of the EM algorithm to estimate the parameters of the model I think it is #1 because at each iteration of the EM algorithm you are estimating the parameters of a single multivariate distribution.

You seem to confuse sample and observations: if one takes $n$ observations from a $p$-dimensional mixture distribution, each observation is a $p$-dimensional vector. It is associated with a latent variable that takes as value 1,2,... depending on which component the vector is generated from.

To make the distinction even clearer, here is a sample of 250 points generated from a univariate mixture$$0.3\mathcal{N}(0,1)+0.7\mathcal{N}(2,1)$$

generated as

data=(rnorm(250)+(runif(250)<.7)*2)


and here s a sample of 250 points generated from a bivariate mixture$$0.3\mathcal{N}_2(0_2,I_2)+0.7\mathcal{N}_2((6,-3),i)$$

generated as

dota=rmvnorm(250,mean=rep(0,2))+(runif(250)<.7)*c(6,-3)


In the first case, the sample starts with

> data[1:10]
[1]  1.1093865  3.9148698  0.7089910  0.8179517  2.1479357  1.1273827
[7]  1.9081127  1.5963952  3.0439887 -0.7480628


and in the second case it begins as

> dota[1:10,]
[,1]       [,2]
[1,]  4.082637 -2.3638552
[2,]  6.417835 -1.9432360
[3,]  5.619073 -0.8438118
[4,]  6.704312 -3.1426273
[5,]  6.000480 -2.8611720
[6,]  5.370948 -2.0932625
[7,] -1.202043  0.7533113
[8,] -0.384621 -0.4188970
[9,]  6.910961 -1.3184499
[10,]  4.453260 -2.9979343

• Yes the .67 will have a latent variable,z, that is 1 or 2 indicating if .67 came from distribution 1 or 2. My question is are the distributions,1 &2, multivariate or univarite....setup #1 or #2 – user3022875 Nov 7 '16 at 15:50
• If 0.67 is one observation this is a one-dimension mixture (setup #1). If 0.67 is a component of an observation vector, for instance, (0.67,2.2), this is a two-dimensional mixture (setup #2). – Xi'an Nov 7 '16 at 16:11
• what if the observation vector has 100 term (.67, 2.2,1.5.......) then you would say there are a 100 dimensional mixture with k=1 ~ N(mu1,mu2, ...mu100, covariace matrix 100x100)? In my example I state this is 1 dimensional vector, X, with 2 components. Components are distributions. – user3022875 Nov 7 '16 at 16:18
• It doesn't matter how many observations X has, 1 or 2 or 100, each observation will only come from 1 of 2 distributions, k=1 or k=2. the distributions do not change from being univariate to multivariate based on the length of X. my question is given 1 vector, i.e. 1 D, and 2 components are the components univariate or multivariate. The author says multivariate and with k=2, that would mean in the EM algorithm you'd have to estimate 4 means and 2 covariance matricies each time through the loop. I believe you only estimate 2 means and 1 covariance matrix in the loop. – user3022875 Nov 7 '16 at 16:21
• The meaning of X itself is unclear. Let me repeat my answer: the dimension of the mixture is the dimension of each observation in the sample. – Xi'an Nov 7 '16 at 16:22