0
$\begingroup$

With reference to the following definition of GMM (see snapshot from Reynolds (1)),

enter image description here

I have two doubts:

In the definition of probability density, the covariance matrix (denoted by sigma) is represented as a vector (since it is represented in bold letters) for each component density. How is it possible? In my opinion, there must be a single covariance matrix (as a scalar and not as a vector) for each component densities.

Secondly, is my interpretation correct when I say that the mean vector (denoted by $mu$) will also be $D$ dimensional if the the data vector $x$ is $D$ dimensional for each component density?

I also think that the order of covariance matrix must be $D\times D$. Correct me if I am wrong.

Please clarify.

(1): Douglas Reynolds,
"Gaussian Mixture Models"
Tutorial paper
MIT Lincoln Laboratory
Lexington, MA

$\endgroup$
  • $\begingroup$ Could you provide a link or a reference from where you have this definition? $\endgroup$ – Nikolas Rieble Sep 23 '16 at 9:46
  • 1
    $\begingroup$ I was reading it from a tutorial paper that I was reading (I may not be able to provide you with the url...However here are the details: TOPIC-Gaussian Mixture Models; AUTHOR- Douglas Reynolds; UNIVERSITY- MIT Lincoln Laboratory, 244 Wood St., Lexington, MA 02140, USA) $\endgroup$ – Upendra Pratap Singh Sep 23 '16 at 9:50
  • $\begingroup$ Do you have a year for the document it's taken from? $\endgroup$ – Glen_b Sep 23 '16 at 9:55
3
$\begingroup$

is represented as a vector (since it is represented in bold letters)

Not so. It's bold, but it's a capital letter, which usually represents a matrix, and it has to be a matrix to be a covariance. There's nothing that forces a bold letter to be a vector rather than a matrix.

Each of the components are themselves multivariate Gaussian, so each has a covariance matrix of dimension $D\times D$. There are $M$ such components.

is my interpretation correct when I say that the mean vector (denoted by $\mu$) will also be $D$ dimensional if the the data vector $x$ is $D$ dimensional for each component density?

Correct -- and consequently the variance-covariance matrix of the $x$'s ($\Sigma_i$) must be $D\times D$ as stated above.

I also think that the order of covariance matrix must be DxD. Correct me if I am wrong.

Correct -- and so it is. But earlier you said:

In my opinion, there must be a single covariance matrix (as a scalar

so you have contradicted yourself there. It can't be a scalar; it's a matrix as you state at the end.

$\endgroup$
  • $\begingroup$ A capital represents a matrix, true and bold represents a vector. Therefore it is vector of matrices. Or to speak in code language: It is an array which contains covariance matrix for distribution i on position i. $\endgroup$ – Nikolas Rieble Sep 23 '16 at 9:46
  • 1
    $\begingroup$ @NikolasRieble -- That's not how it works. $\endgroup$ – Glen_b Sep 23 '16 at 9:47
  • $\begingroup$ You reputation almost conviced me itself. I understand now that mathematically speaking, Sigma is only a covariance matrix and not a vector at all. Yet when implementing it using code, I computed an array of covariance matrices where I used to i-th element for the i-th distribution. This question reminded me of my code, so i mixed up mathematical and code language. You are (as you know) right. $\endgroup$ – Nikolas Rieble Sep 23 '16 at 9:48
1
$\begingroup$

You are right. The bold sigma is a vector of covariance matrices. Yet not the vector is used, but the i-th entry. The same counts for mu - which is a vector of all centers. Yet only the i-th entry is used for the i-th distribution.

Therefore the formula is just fine.

EDIT: Sigma is not a vector, but only a covariance matrix. Implementing algorithms such as EM for GMM (Expectation Maximizization for Gaussian Mixture Models) one tends to (I did) save the covariance matrices in an array and uses the i-th element for the i-th distribution.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.