Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I'm looking for some good terminology to describe what I'm trying to do, to make it easier to look for resources.

So, say I have two clusters of points A and B, each associated to two values, X and Y, and I want to measure the "distance" between A and B - i.e. how likely is it that they were sampled from the same distribution (I can assume that the distributions are normal). For example, if X and Y are correlated in A but not in B, the distributions are different.

Intuitively, I would get the covariance matrix of A, and then look at how likely each point in B is to fit in there, and vice-versa (probably using someting like Mahalanobis distance).

But that is a bit "ad-hoc", and there is probably a more rigorous way of describing this (of course, in practice I have more than two datasets with more than two variables - I'm trying to identify which of my datasets are outliers).


share|improve this question
Hmm, it seems that Mutual Information may be what I'm looking for. – Emile Oct 28 '10 at 13:13
Post it as an answer, you'll gather reputation. – mbq Oct 28 '10 at 13:37
Dunno why, but a Mantel test flashed in front of my eyes when I read your post. – Roman Luštrik Nov 6 '10 at 15:43
up vote 9 down vote accepted

There is also the Kullback-Leibler divergence, which is related to the Hellinger Distance you mention above.

share|improve this answer
Yup, I'm using that, thanks :) – Emile Oct 29 '10 at 8:40
can one calculate the Kullback-Leibler divergence of points without making an assumption of the underlying probability density the points came from ? – Andre Holzner Nov 6 '10 at 16:22

Hmm, the Bhattacharyya distance seems to be what I'm looking for, though the Hellinger distance works too.

share|improve this answer
-- Thank you! -- You're welcome. – Emile Oct 28 '10 at 13:49
Are you enjoying your own personal Q & A? ;-) – Gavin Simpson Oct 28 '10 at 14:19

The most complete survey is provided in Statistical Inference Based on Divergence Measures by Leandro Pardo, Complutense University, Chapman Hall 2006.

share|improve this answer
Welcome to CrossValidated, Mark! – whuber Nov 6 '10 at 15:11


  • Minkowski-form
  • Weighted-Mean-Variance (WMV)

Nonparametric test statistics

  • 2 (Chi Square)
  • Kolmogorov-Smirnov (KS)
  • Cramer/von Mises (CvM)

Information-theory divergences

  • Kullback-Liebler (KL)
  • Jensen–Shannon divergence (metric)
  • Jeffrey-divergence (numerically stable and symmetric)

Ground distance measures

  • Histogram intersection
  • Quadratic form (QF)
  • Earth Movers Distance (EMD)
share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.