Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 1 down vote favorite

It seems to me that it is possible to cluster unidimensional distribution in the space of their parameters, for example, we can try to find similar normal distributions in the space of $(\mu,\sigma)$ pairs.

However, is there a way to cluster multinormal distributions given their means and covariance matrices?

share|improve this question
Because you have received answers that are based on two different interpretations of the question, clarification is needed. Are you attempting to cluster data assumed to be drawn from a multivariate normal mixture or do you (perhaps) have a set of distributions (already given in terms of their multivariate means and covariance matrices) and you wish to cluster them? – whuber♦ Aug 27 '12 at 13:54
1  
The later is true: I'm given parameter sets for distributions. – user829438 Aug 27 '12 at 14:45

2 Answers

up vote 0 down vote accepted

Use the Mahalanobis distance. First you need to decide on how many clusters you want and where the cneters should be. Then put the norml distributions into the cluster whose center is closest based on the Mahalanobis distance. The Mahalanobis distance is the natural distance measure for multivariate normal distributions.

share|improve this answer
This looks like it addresses the intended interpretation of the question. But how does one decide where the "centers should be"? Isn't that the crux of the matter? – whuber♦ Aug 27 '12 at 14:47
@whuber Yes to actually do the clustering you need to specify the number of clusters and the cluster centers. But isn't that really a subjective matter for the user to decide with the possibility of help from some external information about the problem? – Michael Chernick Aug 27 '12 at 14:58
@MichaelChernick, thanks for your answer. However I'm still a bit confused about the way to calculate the 'mean' of the set of distributions that belong to one cluster (for the center update step of the k-means). It seems that it is possible to average the vectors of means, but what should I do with their covariance matrices? – user829438 Aug 28 '12 at 5:21
@user829438 If you are going to assume the covariance matrices are identical pool them in the same way that variance estimates are pooled when they are assumed to be equal in the two-sample problem. – Michael Chernick Aug 28 '12 at 11:01
up vote 1 down vote

Yes. Have a look at the EM algorithm:

https://en.wikipedia.org/wiki/Expectation%E2%80%93maximization_algorithm

It will try to fit a multivariate gaussian mixture distribution to your data, converging to a local minimum. By trying different seeds you have a good chance of finding the global minimum.

share|improve this answer
Could you clarify what you mean by your very last statement? Consider, for example, any Gaussian mixture such that one component (the $j$th, say) has a mean exactly equal to a single data point and we let the standard deviation of that component go to zero. Without even caring which point we've chosen, the likelihood of such a mixture is unbounded (in the limit as $\sigma_j \to 0$)! – cardinal Aug 27 '12 at 15:14
Or even more simply, a single data point. There is a global minimum, where the mean is the point and the variance is 0. Yes, EM will find this (degenerate) global minimum. Probably even with every seed, as there are no other local minima. But assume a data set generated by three gaussian processes, and you are trying to model it using two only. Most likely, there will be multiple stable solutions, aka local minima. One of them is best, but you need multiple runs of EM to find all local minima. – Anony-Mousse Aug 27 '12 at 18:00
I think maybe what I was saying was misunderstood (or I am misunderstanding what you were stating). As long as you are fitting a mixture with at least two components, the likelihood is always unbounded no matter how the data are actually generated. You make it unbounded by fixing one component exactly as we both said: Set its mean to the data point and its variance to zero. So, there is always a local maximum that is infinite. – cardinal Aug 27 '12 at 18:53
1  
Thanks. I know the EM algorithm, though. :-) My point is that your last statement is vacuous. The log-likelihood of any multivariate gaussian mixture is always unbounded. So, you can trivially find a global minimum (without needing any randomness or seeding at all or even any algorithm; just by inspection); but it is essentially meaningless. – cardinal Aug 27 '12 at 21:19
1  
Ah, I see what you mean. I wasn't aware of this problem. Well, maybe you can put in constraints - such as a minimum standard deviation - to avoid the solutions to degenerate to this kind of minima. – Anony-Mousse Aug 28 '12 at 6:07
show 2 more comments

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.